Down? · general · Zulip Chat Archive

Is all of The Changelog down? My pod feed isn't playing the ep from today and the website won't load. Not seeing any mentions on bsky or here though...

Nabeel S (Aug 12 2025 at 01:32):

Ron Waldon-Howe (Aug 12 2025 at 02:21):

Gerhard (Aug 12 2025 at 07:17):

No issues reported by any of the world-wide synthetic probes, and global traffic levels remain unchanged. FWIW https://status.changelog.com/

mray (Aug 12 2025 at 14:58):

I was able to stream the show today but I'm still unable to load up https://changelog.com for some reason. I've done all the usual troubleshooting things like clearing browser cookies/data, and trying with diff devices, etc

Jerod Santo (Aug 12 2025 at 15:17):

I am seeing some error logs in the pipedream app on Fly, but I'm not sure if those are relevant to this

mray (Aug 12 2025 at 15:31):

Jerod Santo (Aug 12 2025 at 15:36):

My guess is it's the pipely instance nearest you, whereas the rest of them are responding a-ok

Gerhard (Aug 12 2025 at 19:47):

I am routing my phone traffic via Dallas (using VPN) and everything works as it should.

Some errors in the logs are normal, there are all sorts of clients, including continuous probing & vulnerability scanners.

I am curious @mray does https://pipedream.changelog.com load for you? Do you get the same behaviour if you use a different browser?

Another thing that would be useful to troubleshoot this is a trace / mtr from where you are having issues connecting.

The last time that I ended up troubleshooting this type of behaviour, the root cause was the ISP that was proxying all traffic. The tell-tale sign was the IP. In the changelog.com case, there are currently two IPs:

Matt Johnson (Aug 12 2025 at 20:15):

I didn't see any problems myself yesterday evening when I saw this reported, but I also didn't check to see where I was connecting to at the time yesterday.

I can check where curl connects to here (which appears to be the Fastly IP Gerhard mentioned):

$ curl -s -o /dev/null -v "https://changelog.com/" 2>&1| head -n 8
* Host changelog.com:443 was resolved.
* IPv6: (none)
* IPv4: 151.101.129.162, 137.66.2.20
*   Trying 151.101.129.162:443...

If I use curl and extract a few of the HTTP headers this is what I'm currently seeing:

$ curl -s -o /dev/null -D - "https://changelog.com/" | grep -i '^http\|^server\|^fly\|^age\|^x-\|^via\|^cache'
HTTP/2 200
cache-control: no-store, must-revalidate
fly-request-id: 01K2FY0C637GMRRYEGWFT9VE14-ord
server: Fly/f2e242947 (2025-08-11)
x-content-type-options: nosniff
x-download-options: noopen
x-frame-options: SAMEORIGIN
x-permitted-cross-domain-policies: none
x-request-id: GFsdcrVmfA1BcBsA6rQC
x-varnish: 1182003 3571763
age: 68
via: 2 fly.io, 2 fly.io, 1.1 82d334c7750248 (Varnish/7.7), 2 fly.io
cache-status: region=ord; origin=app(localhost:5000),changelog-2025-05-05.fly.dev; ttl=-8.176; grace=86400.000; hit; stale; hits=6

$ curl -s -o /dev/null -D - "https://changelog.com/" | grep -i '^http\|^server\|^fly\|^age\|^x-\|^via\|^cache'
HTTP/2 200
cache-control: no-store, must-revalidate
server: Fly/f2e242947 (2025-08-11)
x-changelog-vanity-redirect: false
x-content-type-options: nosniff
x-download-options: noopen
x-frame-options: SAMEORIGIN
x-permitted-cross-domain-policies: none
x-request-id: GFsd06KDBsiKiqUGmOuR
via: 1.1 fly.io, 1.1 fly.io, 1.1 varnish
fly-request-id: 01K2FYAZYJ6FNZ4GCPPHP4QXEE-ord
age: 0
x-served-by: cache-chi-klot8100105-CHI
x-cache: MISS
x-cache-hits: 0
x-timer: S1755029275.568744,VS0,VE92

...so it looks like my requests are typically getting routed to a Chicago edge node in either case.

mray (Aug 13 2025 at 19:22):

@Gerhard It's resolving to 137.66.2.20, 151.101.129.162. But it's working just fine today. idk I'm just going to chalk it up to PEBKAC and shrug. Thanks everyone for troubleshooting and chiming in

Matt Johnson (Aug 15 2025 at 04:01):

While listening to the new Interview episode that dropped today I went to visit the website to view the links in the show notes and everything was working fine for me, but after I was done I decided to browse a little (to do a little testing based on some of the above described behavior) and after visiting a few pages I hit one that just wasn't loading after several seconds, so I forced a refresh and then it loaded... browsed a bit more and had the same behavior where it just hung without loading the page, but the loading wheel was just spinning in chrome... then I pulled up the network tab of the developer tools to try and figure out what was exactly was getting stalled.

I ended up getting it to hang when trying to load the news page, but noticed that it pulled down the document, but was hanging on several static assets. I left the browser and tried running a few curl commands and noticed it would sometimes connect to the Fastly IP, and other times the Pipely IP, but I didn't notice any delays from the shell. Then I noticed the browser window to the side eventually loaded the static assets that were left pending after 8 minutes of waiting.

Gerhard (Aug 15 2025 at 06:44):

Digging into the root cause of this misbehaviour is intriguing. I am off for another week, I only have a phone and can help navigate.

Are you able to reproduce the same behaviour if you force resolve BOTH changelog.com AND cdn.changelog.com to either Fastly or Pipely IP? I am wondering if this is linked to DNS random resolution. For this purpose, being able to see the network waterfall view for the requests that take minutes would be helpful. I am trying to understand which portion of the request is slow for you.

Can you reproduce this in incognito / private browsing window? I am wondering if this is cookie-related (I've noticed "bypass").

Ron Waldon-Howe (Aug 15 2025 at 07:18):

Gerhard (Aug 15 2025 at 08:34):

I've been browsing changelog.com for 5 minutes straight, no slow-downs of any kind. Connecting via a standard 50Mbps Swisscom WiFi.

Matt Johnson (Aug 16 2025 at 02:55):

@Gerhard I tried this morning to modified my system's hosts file to force DNS to resolve (both domains) to only one IP and I noticed that Chrome did not seem to respect that hosts file modification even after clearing the host resolver cache in Chrome. I don't know if I was doing something wrong or not, but in an attempt to only use the Fastly IP in some initial testing, I did noticed that it seemed to be routinely resolving the domains against the Fastly IP at least what I did manage to check.

Morning Testing:
I continued testing with what I observed was typically the Fastly IP:

I was not able to trigger the event from a separate browser (Firefox) in incognito mode.
I was also not able to trigger the event from (Chrome) if I logged out.
Upon logging back in to changelog.com via GitHub I was able to trigger the events again after just a few attempts jumping between the news and home page.

I tried the whole logging out and logging back in a few times to make sure I was seeing consistent results between the two.

I tried to take the triggered network requests from Chrome and Copy as cURL (which includes the cookie values I was using in the browser) so that I could reproduce the effect from a shell, but was unable to reproduce the events from a shell (which I thought was odd and unexpected considering how frequently I could trigger it in the browser).

I was seeing the pending requests in the Chrome browser seem to occur randomly to different requests, sometimes it was the static assets from the cdn and sometimes the document from the root domain.

In my logged in session in Chrome I tried a few different behaviors to see if I could isolate the behavior further.

At this point I had been under the impression that my hosts file was controlling the IP being used (which it might have to some extent ¯\_(ツ)_/¯), but when I tried to switch over to the Pipely IP and quickly confirm the same results there, I noticed that changing my hosts file did not seem to have an effect on the behavior of the browser, and I didn't have anymore time to troubleshoot it further until later in the day.

At the end of my day I picked it back up and was unable to reproduce the behavior or trigger the delayed/pending requests causing the page not to load. I cleared the site data including third party cookies in Chrome, and was still unable to trigger the issue.

From the testing I did perform in the morning, it seemed like the issue may have been cookie related and is likely not affecting most users, so that's a good thing, but I'm afraid I was not able to isolate how to fully reproduce the issue, and there may be other factors at play that I wasn't able to identify. ...so maybe we will just have to wait and see if the scenario appears again later.

Gerhard (Aug 16 2025 at 06:11):

That was super helpful @Matt Johnson as it's hinting at a cookie-related issue, which was one of my suspicions.

The first and most important cookie-related issue that I'm aware of is that we take cookies into account for the assets backend. We need to ignore cookies in VCL for all requests that hit the assets backend. This will resolve some of the issues that you have observed.

As for your DNS setup, you can configure static DNS entries in Chrome. That is the easiest way, as your OS will have it's own DNS cache, and your router will have it's own, so you will need to clear each layer to get DNS to work consistently. If you keep DNS in the browser (each browser exposes this via a special url, e.g. chrome://net-internals/#dns) it will keep things simple.

Stream: general

Topic: Down?

mray (Aug 12 2025 at 01:11):