Skip to content

Conversation

hellais
Copy link
Member

@hellais hellais commented Feb 1, 2024

As part of this PR, everything that is dead code, is removed from the source tree.

This includes:

  • af/ -> this was the old airflow postgresql related code
  • old_backend/ -> this was the super old pre-2016 twisted based backend
  • docs/ -> contained documentation about the ~2016 data pipeline
  • Readme.md -> Still talks about some super old stuff
  • rate_limit_quotas.py -> duplicate of file inside ooniapi/rate_limit_quotas.py
  • ooniapi/scripts -> these scripts expect a postgresql database, which doesn't exist anymore
  • api/refresh_deps -> non-existent files are referenced in it
  • markdown/* -> these docs are not used anywhere and are superseeded by the swagger based docs

@hellais
Copy link
Member Author

hellais commented Feb 2, 2024

It looks like the debops CI tool is not happy that af/prio got deleted, since it's trying to build it:

Traceback (most recent call last):
  File "/__w/backend/backend/./debops-ci", line 5[9](https://github.com/ooni/backend/actions/runs/7752829228/job/21142995136?pr=798#step:8:10)8, in <module>
debops-ci version 2023-11-22.2
    main()
  File "/__w/backend/backend/./debops-ci", line 581, in main
    ci(args)
  File "/__w/backend/backend/./debops-ci", line 297, in ci
Running apt-get update -q
KW: {}
Running apt-get install -qy --no-install-recommends git
KW: {}
Running git merge-base remotes/origin/master HEAD
KW: {}
Running git diff --name-only 48a4634cc3ef93e670c8aae4ac5830[11](https://github.com/ooni/backend/actions/runs/7752829228/job/21142995136?pr=798#step:8:12)8e3c56ee
KW: {}
Building [PosixPath('af/prio'), PosixPath('analysis'), PosixPath('api')]
Running apt-get install -qy --no-install-recommends devscripts
KW: {}
Building package in af/prio
Running dpkg-parsechangelog --show-field Version
KW: {'cwd': PosixPath('af/prio')}
    pkgs_lists = [buildpkg(pd) for pd in pkgdirs]
  File "/__w/backend/backend/./debops-ci", line 297, in <listcomp>
    pkgs_lists = [buildpkg(pd) for pd in pkgdirs]
  File "/__w/backend/backend/./debops-ci", line 209, in buildpkg
    ver = run2("dpkg-parsechangelog --show-field Version", cwd=p)
  File "/__w/backend/backend/./debops-ci", line 80, in run2
    p = run(cmd.split(), capture_output=True, **kw)
  File "/usr/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line [18](https://github.com/ooni/backend/actions/runs/7752829228/job/21142995136?pr=798#step:8:19)[23](https://github.com/ooni/backend/actions/runs/7752829228/job/21142995136?pr=798#step:8:24), in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: PosixPath('af/prio')

We should figure out how to teach it that we don't need that package anymore.

@hellais hellais mentioned this pull request Feb 2, 2024
@hellais
Copy link
Member Author

hellais commented Feb 2, 2024

I have a patch for this inside of ooni/sysadmin#511

This commit should be reverted prior to merge (see 🔄 emoji 🙃)
@hellais
Copy link
Member Author

hellais commented Feb 2, 2024

Ok we are all green, except for the integration tests, which are failing because of the changes to the public list measurements API that are part of the parent PR.

I plan to fix those tests as part of that separate PR.

This branch is ready for review.

@hellais hellais closed this Feb 2, 2024
hellais added a commit to ooni/sysadmin that referenced this pull request Feb 5, 2024
Summary of changes:
* Ignore rebuilding of debian packages that have been deleted
* Add `--dry-run` CLI option to make it easier to locally test

As noted inside of ooni/backend#798, debops-ci
will attempt to build a package even when the package has been deleted.

This happens because it will rebuild any package that it detects by
running:

```
git diff --name-only {commit}
```

and filtering by those names that match the string `debian/changelog`.

However, since it's not filtering by the status, which could also be
that the `debian/changelog` had been deleted, it thinks it should
rebuild also a deleted package.

In this patch I fix that, by ignoring any paths that have the `D`
(deleted) status.

I also add support for a `--dry-run` command line flag, allowing you to
better debug what's going on in non-debian systems.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant