Package creation takes long time when there are many files to be packaged.

[chelarua]

Hello

we (together with @ringods) are using swaddle to create a package from many (~ 2000) of files and it seems its taking a bit long(18-20 mins). I checked a bit and it seems that most of the time is spent in _swaddle_touch_setMtimeAndAtimeAllRecursivelyUnderSimulatedRoot when it's calling fakeroot, which also seems to be doing a sleep on each call. I tried disabling _swaddle_touch_setMtimeAndAtimeAllRecursivelyUnderSimulatedRoot and the packaging time was much shorter, about a minute.

Any suggestions/ advice on this? Thank you

[raphaelcohn]

Hmmm... The touch logic exists in order to ensure that files are packaged with a consistent timestamp when included in a tarball or package. One of the things I've always been keen on is making sure metadata is consistent and useful. It would probably be possible to speed this code up by using find to generate a list of suitable files and then pass these via xargs or find's -exec option, eg something like

swaddle_touch_setMtimeAndAtimeAllRecursivelyUnderSimulatedRoot()
{

    local timestampInEpochSeconds="$(swaddle_configure_timestamp)"
    local mtime="$(date -d @$timestampInEpochSeconds +'%Y%m%d%H%M.%S')"
    local filePath
    for filePath in "$@"
    do
        swaddle_simulateroot_execute find "$filePath" -type f   PASS TO
touch -m -a -c -t "$mtime" "$filePath"
    done
}

_swaddle_touch_setMtimeAndAtimeAllRecursivelyUnderSimulatedRoot()
{
  NOW DEAD
}

Do you want to submit a patch?

Raphael Cohn Chief Architect, StormMQ Co-Chair, OASIS MQTT Standard for the Internet of Things Advisor, UK Government Technical Standards Board https://uk.linkedin.com/in/raphcohn +44 7590 675 756

UK Office: Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom +44 845 3712 567

Registered office: 16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom StormMQ IT Limited is Registered in England and Wales under Company Number 07175657

On 13 October 2015 at 11:29, chelarua [email protected] wrote:

Hello

we (together with @ringods https://github.com/ringods) are using swaddle to create a package from many (~ 2000) of files and it seems its taking a bit long(18-20 mins). I checked a bit and it seems that most of the time is spent in _swaddle_touch_setMtimeAndAtimeAllRecursivelyUnderSimulatedRoot when it's calling fakeroot, which also seems to be doing a sleep on each call. I tried disabling _swaddle_touch_setMtimeAndAtimeAllRecursivelyUnderSimulatedRoot and the packaging time was much shorter, about a minute.

Any suggestions/ advice on this? Thank you

— Reply to this email directly or view it on GitHub https://github.com/raphaelcohn/swaddle/issues/33.

[chelarua]

Hi, thanks for the reply. I tried the suggested implementation and the speed was good, 55 secs instead of 18 minutes with the old implementation.

Made a pull request for this #34.

Thanks

[stormmq]

Right, I'll review the pull req today. Do you want to retain copyright on it? I can add you to the COPYRIGHT file after integrating the pull, or we can make it part of the PR itself.

Raphael Cohn Chief Architect, StormMQ Co-Chair, OASIS MQTT Standard for the Internet of Things Advisor, UK Government Technical Standards Board https://uk.linkedin.com/in/raphcohn +44 7590 675 756

UK Office: Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom +44 845 3712 567

Registered office: 16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom StormMQ IT Limited is Registered in England and Wales under Company Number 07175657

On 28 October 2015 at 10:24, chelarua [email protected] wrote:

Hi, thanks for the reply. I tried the suggested implementation and the speed was good, 55 secs instead of 18 minutes with the old implementation.

Made a pull request for this #34 https://github.com/raphaelcohn/swaddle/pull/34.

Thanks

— Reply to this email directly or view it on GitHub https://github.com/raphaelcohn/swaddle/issues/33#issuecomment-151792852.

[chelarua]

Thanks, don't think that is needed