testsuite require network access by default

[jonassmedegaard]

The test file tests/test_doctests.py fails in a build environment without network access.

Some build environments - including Debian autobuilders - explicitly disable network to make builds more deterministic.

Please rely on network access only optionally. One elegant way to do that is to borrow the logic used in Perl: Check environment variable EXTENDED_TESTING and if not set to a non-empty value then skip all such "extended testing".

Thanks for considering.

[mikeboers]

I'm pretty sure a few more will also fail, because we pull from the FFmpeg FATE suite for samples in a few places.

[jonassmedegaard]

Quoting Mike Boers (2020-04-16 15:23:16)

I'm pretty sure a few more will also fail, because we pull from the FFmpeg FATE suite for samples in a few places.

Right, that file was just the tip of the iceberg.

These files require network during testsuite initialization:

tests/test_doctests.py tests/test_timeout.py

These tests seemingly require network when checked:

TestAudioFifo.test_data
TestCodecContext.test_parse
TestEncoding.test_encoding_aac
TestEncoding.test_encoding_dnxhd
TestEncoding.test_encoding_dvvideo
TestEncoding.test_encoding_h264
TestEncoding.test_encoding_mjpeg
TestEncoding.test_encoding_mp2
TestEncoding.test_encoding_mpeg1video
TestEncoding.test_encoding_pcm_s24le
TestEncoding.test_encoding_png
TestEncoding.test_encoding_tiff
TestContainers.test_context_manager
TestDecode.test_decode_audio_sample_count
TestDecode.test_decoded_motion_vectors
TestDecode.test_decoded_motion_vectors_no_flag
TestDecode.test_decoded_time_base
TestDecode.test_decoded_video_frame_count
TestBasicAudioEncoding.test_audio_transcode
TestAudioProbe.test_container_probing
TestAudioProbe.test_stream_probing
TestDataProbe.test_container_probing
TestDataProbe.test_stream_probing
TestSubtitleProbe.test_container_probing
TestSubtitleProbe.test_stream_probing
TestAudioFifo.test_data
TestCodecContext.test_parse
TestEncoding.test_encoding_aac
TestEncoding.test_encoding_dnxhd
TestEncoding.test_encoding_dvvideo
TestEncoding.test_encoding_h264
TestEncoding.test_encoding_mjpeg
TestEncoding.test_encoding_mp2
TestEncoding.test_encoding_mpeg1video
TestEncoding.test_encoding_mpeg4
TestEncoding.test_encoding_pcm_s24le
TestEncoding.test_encoding_png
TestEncoding.test_encoding_tiff
TestContainers.test_context_manager
TestDecode.test_decode_audio_sample_count
TestDecode.test_decoded_motion_vectors
TestDecode.test_decoded_motion_vectors_no_flag
TestDecode.test_decoded_time_base
TestDecode.test_decoded_video_frame_count
TestBasicAudioEncoding.test_audio_transcode
TestAudioProbe.test_container_probing
TestAudioProbe.test_stream_probing
TestDataProbe.test_container_probing
TestDataProbe.test_stream_probing
TestSubtitleProbe.test_container_probing
TestSubtitleProbe.test_stream_probing
TestVideoProbe.test_container_probing
TestVideoProbe.test_stream_probing
TestPythonIO.test_reading
TestPythonIO.test_reading_no_seek
TestSeek.test_decode_half
TestSeek.test_deprecated_stream_seek
TestSeek.test_seek_end
TestSeek.test_seek_float
TestSeek.test_seek_int64
TestSeek.test_seek_middle
TestSeek.test_seek_start
TestSeek.test_stream_seek
TestStreams.test_selection
TestStreams.test_stream_tuples
TestSubtitle.test_movtext
TestSubtitle.test_vobsub

...and it is an incomplete list - this command still fails:

cd '/build/python-av-7.0.1~git/.pybuild/cpython3_3.8/build'; python3.8 -m pytest --ignore=tests/test_doctests.py --ignore=tests/test_timeout.py -k 'not TestAudioFifo.test_data and not TestCodecContext.test_parse and not TestEncoding.test_encoding_aac and not TestEncoding.test_encoding_dnxhd and not TestEncoding.test_encoding_dvvideo and not TestEncoding.test_encoding_h264 and not TestEncoding.test_encoding_mjpeg and not TestEncoding.test_encoding_mp2 and not TestEncoding.test_encoding_mpeg1video and not TestEncoding.test_encoding_pcm_s24le and not TestEncoding.test_encoding_png and not TestEncoding.test_encoding_tiff and not TestContainers.test_context_manager and not TestDecode.test_decode_audio_sample_count and not TestDecode.test_decoded_motion_vectors and not TestDecode.test_decoded_motion_vectors_no_flag and not TestDecode.test_decoded_time_base and not TestDecode.test_decoded_video_frame_count and not TestBasicAudioEncoding.test_audio_transcode and not TestAudioProbe.test_container_probing and not TestAudioProbe.test_stream_probing and not TestDataProbe.test_container_probing and not TestDataProbe.test_stream_probing and not TestSubtitleProbe.test_container_probing and not TestSubtitleProbe.test_stream_probing and not TestAudioFifo.test_data and not TestCodecContext.test_parse and not TestEncoding.test_encoding_aac and not TestEncoding.test_encoding_dnxhd and not TestEncoding.test_encoding_dvvideo and not TestEncoding.test_encoding_h264 and not TestEncoding.test_encoding_mjpeg and not TestEncoding.test_encoding_mp2 and not TestEncoding.test_encoding_mpeg1video and not TestEncoding.test_encoding_mpeg4 and not TestEncoding.test_encoding_pcm_s24le and not TestEncoding.test_encoding_png and not TestEncoding.test_encoding_tiff and not TestContainers.test_context_manager and not TestDecode.test_decode_audio_sample_count and not TestDecode.test_decoded_motion_vectors and not TestDecode.test_decoded_motion_vectors_no_flag and not TestDecode.test_decoded_time_base and not TestDecode.test_decoded_video_frame_count and not TestBasicAudioEncoding.test_audio_transcode and not TestAudioProbe.test_container_probing and not TestAudioProbe.test_stream_probing and not TestDataProbe.test_container_probing and not TestDataProbe.test_stream_probing and not TestSubtitleProbe.test_container_probing and not TestSubtitleProbe.test_stream_probing and not TestVideoProbe.test_container_probing and not TestVideoProbe.test_stream_probing and not TestPythonIO.test_reading and not TestPythonIO.test_reading_no_seek and not TestSeek.test_decode_half and not TestSeek.test_deprecated_stream_seek and not TestSeek.test_seek_end and not TestSeek.test_seek_float and not TestSeek.test_seek_int64 and not TestSeek.test_seek_middle and not TestSeek.test_seek_start and not TestSeek.test_stream_seek and not TestStreams.test_selection and not TestStreams.test_stream_tuples and not TestSubtitle.test_movtext and not TestSubtitle.test_vobsub'

--

• Jonas Smedegaard - idealist & Internet-arkitekt
• Tlf.: +45 40843136 Website: http://dr.jones.dk/

[x] quote me freely [ ] ask before reusing [ ] keep private

[mikeboers]

Is there a stage in the build environment in which network access is allowed to fetch the assets for the tests?

[jonassmedegaard]

On Debian autobuilders, no. But there is the option of including the test assets in a package - either shipping them with this package or separately packaged. Depending on what really need testing, a more elegant option is, I guess, to not use ffmpeg test assets but tell ffmpeg to generate test data and use that (as I guess what needs testing is not internal routines of ffmpeg but interfacing with those)

[jonassmedegaard]

This seems to be the most minimal workaround for now:

cd '/build/python-av-7.0.1~git/.pybuild/cpython3_3.8/build'; python3.8 -m pytest --ignore=tests/test_doctests.py --ignore=tests/test_timeout.py -k '-test_audio_transcode -test_container_probing -test_context_manager -test_data -test_decode_audio_sample_count -test_decoded_motion_vectors -test_decoded_time_base -test_decoded_video_frame_count -test_decode_half -test_deprecated_stream_seek -test_encoding -test_movtext -test_parse -test_reading -test_seek_end -test_seek_float -test_seek_int64 -test_seek_middle -test_seek_start -test_selection -test_stream_probing -test_stream_seek -test_stream_tuples -test_vobsub'

[jlaine]

Hi @jonassmedegaard ! Are you planning on packaging PyAV for Debian? That would be awesome news!

[jonassmedegaard]

Quoting Jeremy Lainé (2020-04-22 09:51:04)

Hi @jonassmedegaard ! Are you planning on packaging PyAV for Debian? That would be awesome news!

Yes, done already and now just awaiting ftpmaster approval - it is listed as "python-av" in https://ftp-master.debian.org/new.html

--

• Jonas Smedegaard - idealist & Internet-arkitekt
• Tlf.: +45 40843136 Website: http://dr.jones.dk/

[x] quote me freely [ ] ask before reusing [ ] keep private

[jlaine]

@jonassmedegaard I can't remember, does buildd accept network tests which only rely on the loopback interface?

[jonassmedegaard]

Quoting Jeremy Lainé (2020-04-30 16:42:50)

@jonassmedegaard I can't remember, does buildd accept network tests which only rely on the loopback interface?

I think it does, yes - but am not certain - try ask on irc in #debian-devel

--

• Jonas Smedegaard - idealist & Internet-arkitekt
• Tlf.: +45 40843136 Website: http://dr.jones.dk/

[x] quote me freely [ ] ask before reusing [ ] keep private

[jlaine]

@jonassmedegaard how are you handling this in Debian nowadays?

[jonassmedegaard]

During build (where network access is forbidden and blocked) I test like this:

pytest --ignore=tests/test_doctests.py --ignore=tests/test_timeout.py -k 'not test_audio_transcode and not test_codec_tag and not test_container_probing and not test_context_manager and not test_data and not test_decode_audio_sample_count and not test_decoded_motion_vectors and not test_decoded_time_base and not test_decoded_video_frame_count and not test_decode_half and not test_deprecated_stream_seek and not test_encoding and not test_movtext and not test_parse and not test_reading and not test_seek_end and not test_seek_float and not test_seek_int64 and not test_seek_middle and not test_seek_start and not test_selection and not test_stream_probing and not test_stream_seek and not test_stream_tuples and not test_vobsub and not test_reformat_pixel_format_align'

For continuous integration tests I do same, and then run another test flagged as "flaky" and "needs-internet", where I omit only test_reformat_pixel_format_align (which consistently fails on PowerPC).

[jonassmedegaard]

if you want to know the exact tests I omit then it is easier to read here.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jlaine]

Lets keep this alive

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jonassmedegaard]

la la la

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.