snippetdockerMinor
How to generate a list of every external URL accessed during docker build?
Viewed 0 times
duringdockereverygenerateexternalhowlistaccessedbuildurl
Problem
Is there a straight forward way to generate a
Example
An example list might look like this:
(but a real case using an extensive Dockerfile may be more complex and contain many more URLs/IPs).
Things I tried so far
IDSs I attempted to install and use were feature rich and probably more complex than required for what I'm attempting. Configuring IDS engines is not trivial (especially for someone with little network knowledge), so I'd like to explore simpler solutions, if possible.
I captured build logs with this:
and extracted lines containing a URL with this regular expression:
A lot of URLs are found, but... i) it doesn't find URLs which aren't explicitly stated in the build logs, and ii) it finds URLs which aren't actually accessed but which are simply stated in logs (i.e. logs along the lines of "For more info, visit https://www.example.com".
Question
How can I generate a nice clean list of each external URL visited during a
.txt file containing each URL (or corresponding IP) visited during a docker build process?Example
An example list might look like this:
# urls.txt
https://hub.docker.com/
https://www.npmjs.com/(but a real case using an extensive Dockerfile may be more complex and contain many more URLs/IPs).
Things I tried so far
- Installing and using an intrusion detection engine
IDSs I attempted to install and use were feature rich and probably more complex than required for what I'm attempting. Configuring IDS engines is not trivial (especially for someone with little network knowledge), so I'd like to explore simpler solutions, if possible.
- Collecting and parsing docker build logs
I captured build logs with this:
docker build --no-cache --progress=plain . &> build.log
and extracted lines containing a URL with this regular expression:
(http|ftp|https):\/\/([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:\/~+#-]*[\w@?^=%&\/~+#-])
A lot of URLs are found, but... i) it doesn't find URLs which aren't explicitly stated in the build logs, and ii) it finds URLs which aren't actually accessed but which are simply stated in logs (i.e. logs along the lines of "For more info, visit https://www.example.com".
Question
How can I generate a nice clean list of each external URL visited during a
docker build process?Solution
One thing that pops my mind is using either of Wireshark, tshark, or tcpdump for capturing network packets originating at docker daemon while docker build is running.
After recording the traffic, need to filter & process the captured data.
After recording the traffic, need to filter & process the captured data.
Context
StackExchange DevOps Q#17369, answer score: 3
Revisions (0)
No revisions yet.