This is the old XigmaNAS forum in read only mode,
it will taken offline by the end of march 2021!


I like to aks Users and Admins to rewrite/take over important post from here into the new fresh main forum!
Its not possible for us to export from here and import it to the main forum!

Suddenly cannot cache packages with squid - was working 27/12/2017

Jails with XigmaNAS
Forum rules
• Set-Up Guide • FAQs • Forum Rules
Post Reply
nas4me2
NewUser
NewUser
Posts: 9
Joined: 21 Mar 2017 07:47
Status: Offline

Suddenly cannot cache packages with squid - was working 27/12/2017

Post by nas4me2 »

Interesting issue started in the last couple of days with pkg.freebsd.org - the packagesite.txz file is no longer being cached by my proxy (which I built to do exactly that - cache packages locally).

Also I note that suddenly pkg.freebsd.org has become pkgmir.geo.freebsd.org in squid's log.

It was working on 27th Dec when I launched "pkg update -f && pkg upgrade" in the jails:

Jail #1:
27/Dec/2017:13:55:41 +1100 x.x.x.33 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/meta.txz HTTP/1.1" 200 1387 TCP_REFRESH_MODIFIED:HIER_DIRECT
27/Dec/2017:13:56:25 +1100 x.x.x.33 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/packagesite.txz HTTP/1.1" 200 6123394 TCP_REFRESH_MODIFIED:HIER_DIRECT

The package 'descriptors' (meta.txz and packagesite.txz) had been updated and so these were downloaded and cached - note the TCP_REFRESH_UNMODIFIED in each entry below:
27/Dec/2017:13:56:47 +1100 x.x.x.33 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/meta.txz HTTP/1.1" 304 377 TCP_REFRESH_UNMODIFIED:HIER_DIRECT
27/Dec/2017:13:56:47 +1100 x.x.x.33 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/packagesite.txz HTTP/1.1" 304 380 TCP_REFRESH_UNMODIFIED:HIER_DIRECT

Jail #2 - fetches them from squid:
27/Dec/2017:13:57:30 +1100 x.x.x.50 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/meta.txz HTTP/1.1" 200 1394 TCP_REFRESH_UNMODIFIED:HIER_DIRECT
27/Dec/2017:13:57:31 +1100 x.x.x.50 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/packagesite.txz HTTP/1.1" 200 6123401 TCP_REFRESH_UNMODIFIED:HIER_DIRECT
27/Dec/2017:13:57:52 +1100 x.x.x.50 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/meta.txz HTTP/1.1" 304 377 TCP_REFRESH_UNMODIFIED:HIER_DIRECT
27/Dec/2017:13:57:52 +1100 x.x.x.50 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/packagesite.txz HTTP/1.1" 304 380 TCP_REFRESH_UNMODIFIED:HIER_DIRECT

Jail #3 (the proxy itself) - squid (the pkg) was updated in the last line:
27/Dec/2017:13:57:58 +1100 127.0.0.1 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/meta.txz HTTP/1.1" 200 1394 TCP_REFRESH_UNMODIFIED:HIER_DIRECT
27/Dec/2017:13:57:58 +1100 127.0.0.1 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/packagesite.txz HTTP/1.1" 200 6123401 TCP_REFRESH_UNMODIFIED:HIER_DIRECT
27/Dec/2017:13:58:21 +1100 127.0.0.1 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/meta.txz HTTP/1.1" 304 377 TCP_REFRESH_UNMODIFIED:HIER_DIRECT
27/Dec/2017:13:58:21 +1100 127.0.0.1 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/packagesite.txz HTTP/1.1" 304 380 TCP_REFRESH_UNMODIFIED:HIER_DIRECT
27/Dec/2017:13:58:40 +1100 127.0.0.1 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/All/squid-3.5.27_2.txz HTTP/1.1" 200 2218439 TCP_MISS:HIER_DIRECT

Jail #4 - still fetched from squid:
27/Dec/2017:13:59:23 +1100 x.x.x.32 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/meta.txz HTTP/1.1" 200 1394 TCP_REFRESH_UNMODIFIED:HIER_DIRECT
27/Dec/2017:13:59:23 +1100 x.x.x.32 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/packagesite.txz HTTP/1.1" 200 6123401 TCP_REFRESH_UNMODIFIED:HIER_DIRECT
27/Dec/2017:13:59:45 +1100 x.x.x.32 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/meta.txz HTTP/1.1" 304 377 TCP_REFRESH_UNMODIFIED:HIER_DIRECT
27/Dec/2017:13:59:45 +1100 x.x.x.32 - - "GET http://pkg.freebsd.org/FreeBSD:11:amd64/latest/packagesite.txz HTTP/1.1" 304 380 TCP_REFRESH_UNMODIFIED:HIER_DIRECT

Other jails exhibited identical behaviour - that is, they fetched squid's copy.

As can be seen above, one copy was downloaded and cached by the proxy and given to each jail thereafter (including itself since the first pkg update was in another jail).

However, today I see this:

Update the proxy first:
30/Dec/2017:14:33:51 +1100 127.0.0.1 - - "GET http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/latest/meta.txz HTTP/1.1" 200 1388 TCP_MISS:HIER_DIRECT
30/Dec/2017:14:34:38 +1100 127.0.0.1 - - "GET http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/latest/packagesite.txz HTTP/1.1" 200 6121983 TCP_MISS:HIER_DIRECT

The TCP_MISS above is not unusual IF the files have changed.

However, they should be fetched from the cache in the lines below - clearly they are not.

30/Dec/2017:14:34:51 +1100 127.0.0.1 - - "GET http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/latest/meta.txz HTTP/1.1" 304 371 TCP_MISS:HIER_DIRECT
30/Dec/2017:14:34:52 +1100 127.0.0.1 - - "GET http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/latest/packagesite.txz HTTP/1.1" 304 374 TCP_MISS:HIER_DIRECT

And in every jail - TCP_MISS remains and the files are downloaded again:
30/Dec/2017:14:35:37 +1100 x.x.x.33 - - "GET http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/latest/meta.txz HTTP/1.1" 200 1388 TCP_MISS:HIER_DIRECT
30/Dec/2017:14:36:23 +1100 x.x.x.33 - - "GET http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/latest/packagesite.txz HTTP/1.1" 200 6121983 TCP_MISS:HIER_DIRECT

And trying again in the proxy - it downloads it AGAIN!!!!!!! (and again and again and again as often as you run it)
30/Dec/2017:15:39:56 +1100 127.0.0.1 - - "GET http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/latest/meta.txz HTTP/1.1" 200 1388 TCP_MISS:HIER_DIRECT
30/Dec/2017:15:40:34 +1100 127.0.0.1 - - "GET http://pkgmir.geo.freebsd.org/FreeBSD:11:amd64/latest/packagesite.txz HTTP/1.1" 200 6121983 TCP_MISS:HIER_DIRECT

Is it me or has something been broken with pkg.freebsd.org?

I haven't touched the pkg conf files recently - in fact the only change was to add the proxy config to /usr/local/etc/pkg.conf as below (obviously this is for the proxy itself):
pkg_env: {
http_proxy: "http://127.0.0.1:3128"
}

/etc/pkg/FreeBSD.conf is unmodified from install:
# $FreeBSD: releng/11.1/etc/pkg/FreeBSD.conf 320745 2017-07-06 17:22:33Z gjb $
#
# To disable this repository, instead of modifying or removing this file,
# create a /usr/local/etc/pkg/repos/FreeBSD.conf file:
#
# mkdir -p /usr/local/etc/pkg/repos
# echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
#

FreeBSD: {
url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",
mirror_type: "srv",
signature_type: "fingerprints",
fingerprints: "/usr/share/keys/pkg",
enabled: yes
}

So, is it me, my config (jail, squid?) or did someone break the pkg system?
Top
nas4me2
NewUser
NewUser
Posts: 9
Joined: 21 Mar 2017 07:47
Status: Offline

SOLVED Re: Suddenly cannot cache packages with squid - was working 27/12/2017

Post by nas4me2 »

Well, after much digging it should have been obvious:

Working:
27/Dec/2017:13:59:45 +1100 x.x.x.32 - - "GET http://pkg.freebsd.org ...

NOT Working:
30/Dec/2017:14:33:51 +1100 127.0.0.1 - - "GET http://pkgmir.geo.freebsd.org ...

So some bright spark decided to randomly implement a new DNS name for the package servers - why this was done is beyond me.

From this link the hint became more obvious:
https://github.com/freebsd/pkg/issues/1 ... t-75430457

In my squid.conf I had this:
refresh_pattern ^http://pkg.freebsd.org/.*\.txz 1440 100% 10080 ignore-private ignore-must-revalidate override-expire ignore-no-cacher

...which of course does NOT map to the new DNS names (sigh).

Adding the below line fixed it.
refresh_pattern ^http://pkgmir.geo.freebsd.org/.*\.txz 1440 100% 10080 ignore-private ignore-must-revalidate override-expire ignore-no-cache

Working now...
17/Jan/2018:13:35:25 +1100 10.10.50.37 - - [17/Jan/2018:13:35:25 +1100] "GET http://pkgmir.geo.freebsd.org .... TCP_REFRESH_UNMODIFIED:HIER_DIRECT
Top
Post Reply

Return to “Jails”