Change only diff, hard link existing #4

New issue

Open

opened 2026-08-02 14:57:39 -04:00 by javif89 · 0 comments

javif89 commented

2026-08-02 14:57:39 -04:00

Owner

On normal SSD this tool is fast enough, but over NAS storage it still takes a few seconds.

I cannot fix this for first deploy, but I can make subsequent deploys faster by:

Maintaining a manifest of path -> hash for each file in a deployed site

Then on next deploy:

enum FileWriteJob {
    Write { path: PathBuf, content: Vec<u8> },
    Link  { from: PathBuf, to: PathBuf },
}

Producer:

let hash = hex::encode(Sha256::digest(&content));

match previous.get(&rel) {
    Some(prev) if *prev == hash => {
        w_pool.enqueue(Link { from: live.join(&rel), to: staging.join(&rel) })
    }
    _ => w_pool.enqueue(Write { path: staging.join(&rel), content }),
}

manifest.insert(rel, hash);   // persist after a successful deploy

Worker:

match job {
    Write { path, content } => fs::write(&path, content)?,
    Link  { from, to }      => fs::hard_link(&from, &to)?,
}

This way unchanged files don't have to go through the network boundary. I'm already
reading the bytes for each file so this operation is cheap.

Things to remember:

Store the relative path instead of the full path in the manifest so we can handle
the folder being moved to a different parent. Maybe /sites -> /another-site-dir type deal.

On normal SSD this tool is fast enough, but over NAS storage it still takes a few seconds. I cannot fix this for first deploy, but I can make subsequent deploys faster by: 1. Maintaining a manifest of path -> hash for each file in a deployed site Then on next deploy: ```rust enum FileWriteJob { Write { path: PathBuf, content: Vec<u8> }, Link { from: PathBuf, to: PathBuf }, } Producer: let hash = hex::encode(Sha256::digest(&content)); match previous.get(&rel) { Some(prev) if *prev == hash => { w_pool.enqueue(Link { from: live.join(&rel), to: staging.join(&rel) }) } _ => w_pool.enqueue(Write { path: staging.join(&rel), content }), } manifest.insert(rel, hash); // persist after a successful deploy Worker: match job { Write { path, content } => fs::write(&path, content)?, Link { from, to } => fs::hard_link(&from, &to)?, } ``` This way unchanged files don't have to go through the network boundary. I'm already reading the bytes for each file so this operation is cheap. Things to remember: - Store the relative path instead of the full path in the manifest so we can handle the folder being moved to a different parent. Maybe /sites -> /another-site-dir type deal.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

javif89/static-forge#4

No description provided.

Rows
Columns