Continuously stuck in the "Running" state

[Zty031220]

Dear Codabench team

I am the organizer of the following challenge: https://www.codabench.org/competitions/13553/. Now it's my turn to submit. It has been stuck in the "Running" state and has been like this for a whole day.

[Didayolo]

Dear @Zty031220,

Can you please try re-submitting? You may have made your submissions right during the platform upgrade.

[Zty031220]

Dear @Zty031220,

Can you please try re-submitting? You may have made your submissions right during the platform upgrade.

I have just submitted a new one here, and it is still running for over an hour.

[Didayolo]

I re-run a submission and it failed on the scoring worker with the following logs.

2026-02-11 13:09:08.030 | INFO     | celery.worker.strategy:task_message_handler:161 - Task compute_worker_run[e89882d1-a353-4a5f-ae8c-7d277490de60] received
2026-02-11 13:09:08.032 | INFO     | compute_worker:run_wrapper:217 - Received run arguments: 
 {"user_pk": 51436,
 "submissions_api_url": "https://www.codabench.org/api",
 "secret": "2715221a-c37f-49db-a8ca-3e4ebb5f3296",
 "docker_image": "george200150/ntire26-cvlai:latest",
 "execution_time_limit": 600,
 "id": 536918,
 "is_scoring": true,
 "prediction_result": "https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/prediction_result/2026-02-11-1770815244/2a9fcc2e1660/prediction_result.zip?<HIDE>",
 "scoring_result": "https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/scoring_result/2026-02-11-1770815244/60e5cd8eef64/scoring_result.zip?<HIDE>",
 "reference_data": "https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/dataset/2026-02-03-1770148079/a7696bf62bf8/reference_data_1.zip?<HIDE>",
 "ingestion_only_during_scoring": false,
 "program_data": "https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/dataset/2026-02-04-1770238163/3f53c99a1fe4/scoring_program_2026.zip?<HIDE>",
 "scoring_stdout": "https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/submission_details/2026-02-11-1770815265/d3ee218f8ac3/scoring_stdout.txt?<HIDE>",
 "scoring_stderr": "https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/submission_details/2026-02-11-1770815265/dbb33a0bcef4/scoring_stderr.txt?<HIDE>",
 "scoring_ingestion_stdout": "https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/submission_details/2026-02-11-1770815265/3f5f905e3031/scoring_ingestion_stdout.txt?A<HIDE>",
 "scoring_ingestion_stderr": "https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/submission_details/2026-02-11-1770815266/126df158f072/scoring_ingestion_stderr.txt?<HIDE>"
}
2026-02-11 13:09:08.032 | INFO     | compute_worker:_prep_cache_dir:1061 - Checking if cache directory needs to be pruned...
2026-02-11 13:09:08.033 | INFO     | compute_worker:_prep_cache_dir:1066 - Cache directory does not need to be pruned!
2026-02-11 13:09:08.033 | INFO     | compute_worker:_get_bundle:595 - Getting bundle https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/dataset/2026-02-04-1770238163/3f53c99a1fe4/scoring_program_2026.zip?<HIDE> to unpack @ program
2026-02-11 13:09:08.126 | INFO     | compute_worker:_get_bundle:595 - Getting bundle https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/dataset/2026-02-03-1770148079/a7696bf62bf8/reference_data_1.zip?A<HIDE> to unpack @ input/ref
2026-02-11 13:09:17.724 | INFO     | compute_worker:_get_bundle:595 - Getting bundle https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/prediction_result/2026-02-11-1770815244/2a9fcc2e1660/prediction_result.zip?<HIDE> to unpack @ input/res
2026-02-11 13:09:17.757 | WARNING  | compute_worker:_get_bundle:633 - Failed. Retrying in 60 seconds...
2026-02-11 13:10:29.185 | INFO     | compute_worker:_get_container_image:509 - Running pull for image: george200150/ntire26-cvlai:latest
2026-02-11 13:10:29.843 | INFO     | compute_worker:show_progress:89 - Status: Image is up to date for george200150/ntire26-cvlai:latest

2026-02-11 13:10:29.847 | INFO     | compute_worker:_update_submission:478 - Updating submission @ https://www.codabench.org/api/submissions/536918/ with data = {'status': 'Running', 'status_details': 'scoring_hostname-worker5', 'secret': '2715221a-c37f-49db-a8ca-3e4ebb5f3296'}
2026-02-11 13:10:30.079 | INFO     | compute_worker:_update_submission:482 - Submission updated successfully!
2026-02-11 13:10:30.079 | INFO     | compute_worker:start:1125 - Running scoring program, and then ingestion program
2026-02-11 13:10:30.079 | INFO     | compute_worker:_run_program_directory:831 - Metadata path is /codabench/tmpfltp6_40/program/metadata
2026-02-11 13:10:30.091 | INFO     | compute_worker:_run_program_directory:835 - Metadata contains:
 {'command': 'python $program/evaluation.py $input $output', 'description': 'Example competition evaluation program.'}
2026-02-11 13:10:30.092 | INFO     | compute_worker:_run_program_directory:975 - Running container with command python /app/program/evaluation.py /app/input /app/output
2026-02-11 13:10:30.778 | INFO     | compute_worker:_run_program_directory:992 - Volume configuration of the container: 
{
│   '/codabench/tmpfltp6_40/program': {'bind': '/app/program', 'mode': 'z'},
│   '/codabench/tmpfltp6_40/output': {'bind': '/app/output', 'mode': 'z'},
│   '/codabench/data': {'bind': '/app/data', 'mode': 'ro'},
│   '/codabench/tmpfltp6_40/shared': {'bind': '/app/shared'},
│   '/codabench/tmpfltp6_40/input': {'bind': '/app/input'}
}
2026-02-11 13:10:30.780 | WARNING  | compute_worker:_run_program_directory:806 - /codabench/tmpfltp6_40/ingestion_program not found, no program to execute
2026-02-11 13:10:37.457 | INFO     | compute_worker:_run_container_engine_cmd:700 - EVALUATION FAILED: ref/ and res/ PNG filenames mismatch and could not be paired.
Missing in res (first 20): ['0000001.png', '0000002.png', '0000003.png', '0000004.png', '0000005.png', '0000006.png', '0000007.png', '0000008.png', '0000009.png', '0000010.png', '0000011.png', '0000012.png', '0000013.png', '0000014.png', '0000015.png', '0000016.png', '0000017.png', '0000018.png', '0000019.png', '0000020.png']
Extra in res (first 20): []
Counts: ref=200 res=0

2026-02-11 13:10:37.458 | ERROR    | compute_worker:_run_container_engine_cmd:709 - Traceback (most recent call last):

2026-02-11 13:10:37.458 | ERROR    | compute_worker:_run_container_engine_cmd:709 -   File "/app/program/evaluation.py", line 358, in <module>
    main()
  File "/app/program/evaluation.py", line 318, in main
    pairs = build_pairs(ref_pngs, res_pngs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/program/evaluation.py", line 277, in build_pairs
    raise RuntimeError(
RuntimeError: ref/ and res/ PNG filenames mismatch and could not be paired.
Missing in res (first 20): ['0000001.png', '0000002.png', '0000003.png', '0000004.png', '0000005.png', '0000006.png', '0000007.png', '0000008.png', '0000009.png', '0000010.png', '0000011.png', '0000012.png', '0000013.png', '0000014.png', '0000015.png', '0000016.png', '0000017.png', '0000018.png', '0000019.png', '0000020.png']
Extra in res (first 20): []
Counts: ref=200 res=0

2026-02-11 13:10:37.857 | INFO     | compute_worker:start:1202 - [exited with 1]
2026-02-11 13:10:37.857 | INFO     | compute_worker:start:1207 - [stdout]
b"EVALUATION FAILED: ref/ and res/ PNG filenames mismatch and could not be paired.\nMissing in res (first 20): ['0000001.png', '0000002.png', '0000003.png', '0000004.png', '0000005.png', '0000006.png', '0000007.png', '0000008.png', '0000009.png', '0000010.png', '0000011.png', '0000012.png', '0000013.png', '0000014.png', '0000015.png', '0000016.png', '0000017.png', '0000018.png', '0000019.png', '0000020.png']\nExtra in res (first 20): []\nCounts: ref=200 res=0\n"
2026-02-11 13:10:37.857 | INFO     | compute_worker:_put_file:1042 - Putting raw data b"EVALUATION FAILED: ref/ and res/ PNG filenames mismatch and could not be paired.\nMissing in res (first 20): ['0000001.png', '0000002.png', '0000003.png', '0000004.png', '0000005.png', '0000006.png', '0000007.png', '0000008.png', '0000009.png', '0000010.png', '0000011.png', '0000012.png', '0000013.png', '0000014.png', '0000015.png', '0000016.png', '0000017.png', '0000018.png', '0000019.png', '0000020.png']\nExtra in res (first 20): []\nCounts: ref=200 res=0\n" in https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/submission_details/2026-02-11-1770815265/d3ee218f8ac3/scoring_stdout.txt?<HIDE>
2026-02-11 13:10:37.891 | INFO     | compute_worker:_put_file:1054 - *** PUT RESPONSE: ***
2026-02-11 13:10:37.891 | INFO     | compute_worker:_put_file:1055 - response: <Response [200]>
2026-02-11 13:10:37.891 | INFO     | compute_worker:_put_file:1056 - content: b''
2026-02-11 13:10:37.891 | INFO     | compute_worker:start:1207 - [stderr]
b'Traceback (most recent call last):\n  File "/app/program/evaluation.py", line 358, in <module>\n    main()\n  File "/app/program/evaluation.py", line 318, in main\n    pairs = build_pairs(ref_pngs, res_pngs)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/app/program/evaluation.py", line 277, in build_pairs\n    raise RuntimeError(\nRuntimeError: ref/ and res/ PNG filenames mismatch and could not be paired.\nMissing in res (first 20): [\'0000001.png\', \'0000002.png\', \'0000003.png\', \'0000004.png\', \'0000005.png\', \'0000006.png\', \'0000007.png\', \'0000008.png\', \'0000009.png\', \'0000010.png\', \'0000011.png\', \'0000012.png\', \'0000013.png\', \'0000014.png\', \'0000015.png\', \'0000016.png\', \'0000017.png\', \'0000018.png\', \'0000019.png\', \'0000020.png\']\nExtra in res (first 20): []\nCounts: ref=200 res=0\n'
2026-02-11 13:10:37.891 | INFO     | compute_worker:_put_file:1042 - Putting raw data b'Traceback (most recent call last):\n  File "/app/program/evaluation.py", line 358, in <module>\n    main()\n  File "/app/program/evaluation.py", line 318, in main\n    pairs = build_pairs(ref_pngs, res_pngs)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/app/program/evaluation.py", line 277, in build_pairs\n    raise RuntimeError(\nRuntimeError: ref/ and res/ PNG filenames mismatch and could not be paired.\nMissing in res (first 20): [\'0000001.png\', \'0000002.png\', \'0000003.png\', \'0000004.png\', \'0000005.png\', \'0000006.png\', \'0000007.png\', \'0000008.png\', \'0000009.png\', \'0000010.png\', \'0000011.png\', \'0000012.png\', \'0000013.png\', \'0000014.png\', \'0000015.png\', \'0000016.png\', \'0000017.png\', \'0000018.png\', \'0000019.png\', \'0000020.png\']\nExtra in res (first 20): []\nCounts: ref=200 res=0\n' in https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/submission_details/2026-02-11-1770815265/dbb33a0bcef4/scoring_stderr.txt?<HIDE>
2026-02-11 13:10:37.904 | INFO     | compute_worker:_put_file:1054 - *** PUT RESPONSE: ***
2026-02-11 13:10:37.904 | INFO     | compute_worker:_put_file:1055 - response: <Response [200]>
2026-02-11 13:10:37.904 | INFO     | compute_worker:_put_file:1056 - content: b''
2026-02-11 13:10:37.905 | INFO     | compute_worker:start:1211 - Program finished
2026-02-11 13:10:37.905 | INFO     | compute_worker:_update_submission:478 - Updating submission @ https://www.codabench.org/api/submissions/536918/ with data = {'status': 'Finished', 'status_details': None, 'secret': '2715221a-c37f-49db-a8ca-3e4ebb5f3296'}
2026-02-11 13:10:38.054 | INFO     | compute_worker:_update_submission:482 - Submission updated successfully!
2026-02-11 13:10:38.054 | INFO     | compute_worker:_update_submission:478 - Updating submission @ https://www.codabench.org/api/submissions/536918/ with data = {'status': 'Failed', 'status_details': 'Could not find scores file, did the scoring program output it?', 'secret': '2715221a-c37f-49db-a8ca-3e4ebb5f3296'}
2026-02-11 13:10:38.222 | INFO     | compute_worker:_update_submission:482 - Submission updated successfully!
2026-02-11 13:10:38.222 | INFO     | compute_worker:clean_up:1292 - Destroying submission temp dir: /codabench/tmpfltp6_40
2026-02-11 13:10:38.486 | INFO     | celery.app.trace:info:131 - Task compute_worker_run[e89882d1-a353-4a5f-ae8c-7d277490de60] succeeded in 90.45414157956839s: None

[Didayolo]

@Zty031220 It looks like the problem is the format of your submission.

It has a directory structure like this:

submission.zip
|- NTIRE/
|---|- image1.png
|---|- image2.png
[...]

Instead of:

submission.zip
|- image1.png
|- image2.png
[...]

The fact that the submission blocks in Running instead of marking Failed and giving relevant logs is independent, and we are working on solving this problem. We need to merge #2030 soon to make this clearer.

[Didayolo]

Now the compute worker crashes during scoring and the submission get stuck in Running (#1205):

2026-02-11 17:58:58.992 | INFO     | compute_worker:_get_bundle:595 - Getting bundle https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/dataset/2026-02-04-1770238163/3f53c99a1fe4/scoring_program_2026.zip?<HIDE> to unpack @ program
2026-02-11 17:58:59.055 | INFO     | compute_worker:_get_bundle:595 - Getting bundle https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/dataset/2026-02-03-1770148079/a7696bf62bf8/reference_data_1.zip?<HIDE> to unpack @ input/ref
2026-02-11 17:59:08.347 | INFO     | compute_worker:_get_bundle:595 - Getting bundle https://miniodis-rproxy.lisn.upsaclay.fr/coda-v2-prod-private/prediction_result/2026-02-11-1770832639/99b86bec00c6/prediction_result.zip?<HIDE> to unpack @ input/res
2026-02-11 17:59:08.383 | WARNING  | compute_worker:_get_bundle:633 - Failed. Retrying in 60 seconds...
2026-02-11 18:00:22.185 | INFO     | compute_worker:_get_container_image:509 - Running pull for image: george200150/ntire26-cvlai:latest
2026-02-11 18:00:22.871 | INFO     | compute_worker:show_progress:89 - Status: Image is up to date for george200150/ntire26-cvlai:latest

2026-02-11 18:00:22.873 | INFO     | compute_worker:_update_submission:478 - Updating submission @ https://www.codabench.org/api/submissions/537336/ with data = {'status': 'Running', 'status_details': 'scoring_hostname-worker3', 'secret': 'f73551a2-1992-428a-ab29-177bff7110f3'}
2026-02-11 18:00:23.155 | INFO     | compute_worker:_update_submission:482 - Submission updated successfully!
2026-02-11 18:00:23.155 | INFO     | compute_worker:start:1125 - Running scoring program, and then ingestion program
2026-02-11 18:00:23.156 | INFO     | compute_worker:_run_program_directory:831 - Metadata path is /codabench/tmp0x9pagf4/program/metadata
2026-02-11 18:00:23.157 | INFO     | compute_worker:_run_program_directory:835 - Metadata contains:
 {'command': 'python $program/evaluation.py $input $output', 'description': 'Example competition evaluation program.'}
2026-02-11 18:00:23.157 | INFO     | compute_worker:_run_program_directory:975 - Running container with command python /app/program/evaluation.py /app/input /app/output
2026-02-11 18:01:23.221 | WARNING  | compute_worker:_run_program_directory:806 - /codabench/tmp0x9pagf4/ingestion_program not found, no program to execute
2026-02-11 18:01:23.222 | INFO     | compute_worker:clean_up:1292 - Destroying submission temp dir: /codabench/tmp0x9pagf4
2026-02-11 18:01:23.790 | ERROR    | celery.app.trace:_log_error:265 - Task compute_worker_run[8f11f73c-bc71-4ba6-b5a4-d889132ad774] raised unexpected: ReadTimeout(ReadTimeoutError("UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)"))
Traceback (most recent call last):

  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
               │    └ <function HTTPConnection.getresponse at 0x7f07344e3820>
               └ <UnixHTTPConnection(host='localhost', port=80) at 0x7f0733f181c0>
  File "/usr/lib/python3.9/site-packages/urllib3/connection.py", line 571, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
    │        └ <function HTTPResponse.begin at 0x7f0735a59b80>
    └ <http.client.HTTPResponse object at 0x7f0733f18580>
  File "/usr/lib64/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
                              │    └ <function HTTPResponse._read_status at 0x7f0735a59af0>
                              └ <http.client.HTTPResponse object at 0x7f0733f18580>
  File "/usr/lib64/python3.9/http/client.py", line 281, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               │    │           └ 65536
               │    └ None
               └ <http.client.HTTPResponse object at 0x7f0733f18580>
  File "/usr/lib64/python3.9/socket.py", line 716, in readinto
    return self._sock.recv_into(b)
           │    │               └ <memory at 0x7f0733bdee80>
           │    └ None
           └ <socket.SocketIO object at 0x7f0733f18460>

socket.timeout: timed out


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 644, in send
    resp = conn.urlopen(
           │    └ <function HTTPConnectionPool.urlopen at 0x7f07344eb670>
           └ <docker.transport.unixconn.UnixHTTPConnectionPool object at 0x7f0733d0bf70>
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              │       └ <function Retry.increment at 0x7f0734542790>
              └ Retry(total=0, connect=None, read=False, redirect=None, status=None)
  File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 474, in increment
    raise reraise(type(error), error, _stacktrace)
          │            │       │      └ <traceback object at 0x7f0733cdbb40>
          │            │       └ ReadTimeoutError("UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)")
          │            └ ReadTimeoutError("UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)")
          └ <function reraise at 0x7f073455a0d0>
  File "/usr/lib/python3.9/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
          └ None
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               │    └ <function HTTPConnectionPool._make_request at 0x7f07344eb4c0>
               └ <docker.transport.unixconn.UnixHTTPConnectionPool object at 0x7f0733d0bf70>
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
    │    │                         │                  └ 60
    │    │                         └ '/v1.47/containers/create?name=e8c685d3-4592-4fee-9169-02fd5292ae55'
    │    └ <function HTTPConnectionPool._raise_timeout at 0x7f07344eb430>
    └ <docker.transport.unixconn.UnixHTTPConnectionPool object at 0x7f0733d0bf70>
  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
          └ <class 'urllib3.exceptions.ReadTimeoutError'>

urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

> File "/usr/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)

  File "//compute_worker.py", line 222, in run_wrapper
    run.start()

  File "//compute_worker.py", line 1137, in start
    loop.run_until_complete(gathered_tasks)

  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()

  File "//compute_worker.py", line 981, in _run_program_directory
    container = client.create_container(

  File "/usr/lib/python3.9/site-packages/docker/api/container.py", line 440, in create_container
    return self.create_container_from_config(config, name, platform)
  File "/usr/lib/python3.9/site-packages/docker/api/container.py", line 456, in create_container_from_config
    res = self._post_json(u, data=config, params=params)
  File "/usr/lib/python3.9/site-packages/docker/api/client.py", line 303, in _post_json
    return self._post(url, data=json.dumps(data2), **kwargs)
  File "/usr/lib/python3.9/site-packages/docker/utils/decorators.py", line 44, in inner
    return f(self, *args, **kwargs)
  File "/usr/lib/python3.9/site-packages/docker/api/client.py", line 242, in _post
    return self.post(url, **self._set_request_timeout(kwargs))
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)