I recently learned about Stream data and how to feed it directly into the PDF viewer I am using. The solution was fairly easy, but it took me a bit to find it and understand what was happening.
A little context
On this project I am using NextJS, MinIO (To access my s3 Buckets), and React PDF Viewer. I'm using NextJS at it's base setup for the api, so that means I'm using the inherent api directory in pages. I am very fond of the auto-magic routing system they've got going on in the background there. Now you are probably wondering why I need to feed the data directly from the API. The S3 Bucket for this project was configured to only be available within our K8s(Kubernetes) cluster, this means I could not use a pre-signed URL to fetch the document from the browser because the browser is outside or the cluster. Thus I needed a way to retrieve the Data from within the cluster. This lead to serving up the data directly from the API.
Sample NextJS Structure
├─ app
├── components
├── lib <-- custom
│ ├── **/*.ts
│ ├── minio.ts
└── pages
├── api
│ ├──[documentId].ts
└── index.tsx
MinIO
I am working with the MinIO JavaScript Client SDK, so I can make my MinIO connections in the API.
I first set up a MinIo config file that has my client definition where I feed in environment variables to connect to my AWS S3 bucket and the methods I need for my app.
import * as Minio from 'minio';
import config from 'lib/config';
import { Readable } from 'stream';
import { NextApiResponse } from 'next';
const { BUCKET_NAME, BUCKET_HOST, BUCKET_PORT, AWS_ACCESSKEY_ID, AWS_SECRET_ACCESS_KEY } = config;
export default function minio(): Minio.Client {
global.minio =
global.minio ||
new Minio.Client({
endPoint: BUCKET_HOST,
port: Number(BUCKET_PORT),
useSSL: false,
accessKey: AWS_ACCESSKEY_ID,
secretKey: AWS_SECRET_ACCESS_KEY,
});
return global.minio;
}
// Checks if Bucket exists, create bucket if it does not
export const upsertMinioBucket = async (minioClient: Minio.Client) => {
const bucketExists = await minioClient.bucketExists(BUCKET_NAME);
if (bucketExists) {
return true;
} else {
await minioClient.makeBucket(BUCKET_NAME, 'us-east-1');
return await minioClient.bucketExists(BUCKET_NAME);
}
};
// Upload file to bucket and returns with object => (err| objInfo)
// Uploads contents from a file to objectName.
// fPutObject(bucketName, objectName, filePath, metaData[, callback])
export const upsertMinioFile = async (minioClient: Minio.Client, filePath: string, fileName: string) => {
if (filePath) {
const objectMade = await minioClient.fPutObject(BUCKET_NAME, fileName, filePath, {});
return objectMade;
}
};
export const loadMinioStream = async (minioClient: Minio.Client, fileName: string | string[]) => {
const response = await minioClient.getObject(BUCKET_NAME, fileName).then((err, stream) => (!err ? stream : err));
return response;
};
export const streamToResponse = async (stream: Readable, res: NextApiResponse): Promise<any> => {
return new Promise<any>(() => {
stream.on('data', function (chunk) {
res.write(chunk);
});
stream.on('end', () => {
res.end();
});
stream.on('error', (err) => res.end(err));
});
};
getObject+ streamToResponse = ❤️
In the API route [documentId].ts, I use the MinIO Object operation getObject and custom method for for running through the stream data to write the stream directly to the response.
import { NextApiRequest, NextApiResponse } from 'next';
import { runMiddleware } from 'lib/middleware';
import CORS from 'cors';
import minio, { loadMinioStream, streamToResponse, upsertMinioBucket } from 'lib/minio';
const client = minio();
// Initializing the cors middleware
const cors = runMiddleware(
CORS({
methods: ['GET', 'POST', 'OPTIONS'],
})
);
export default async function handler(_req: NextApiRequest, res: NextApiResponse) {
await cors(_req, res);
const { documentId = '1' } = _req.query;
const resp = await getDocumentStream(documentId, res);
res.json(resp);
}
export const getDocumentStream = async (documentId: string | string[], res: NextApiResponse) => {
try {
const exists = process.env.NODE_ENV !== 'test' ? await upsertMinioBucket(client) : false;
let resolve: any = null;
if (exists && process.env.NODE_ENV !== 'test') {
const stream = await loadMinioStream(client, documentId);
resolve = await streamToResponse(stream, res);
}
return {
statusCode: 200,
data: resolve,
};
} catch (e) {
return {
statusCode: 500,
data: {
success: false,
error: `${e}`,
},
};
}
};
Feeding into React PDF Viewer
So then to get my file to show up into the PDF viewer, I feed it my API route as the fileURL, and Blam it's there.
export default function PDFViewer({ document}){
...
const [filePath, setFilePath] = useState('');
const workerUrl = 'https://unpkg.com/pdfjs-dist@2.14.305/legacy/build/pdf.worker.js';
useEffect(() => {
if (document) {
let path = `//${window.location.hostname}${port}/api/documents/pdf/${document?.id}`;
const port = window.location.port == '80' ? '' : ':' + window.location.port;
setFilePath(path);
}
}, [document]);
}
return(
<Worker workerUrl={workerUrl}>
<Viewer fileUrl={filePath} />
</Worker>
)
Note: This particular solution is not specific to a PDF, it could be used in cases of images, binary, etc.
If you have made it to this point, I would like to say congrats! You made it to the end! As a reward I present to you this gif!
Top comments (2)
We have only
[documentId].jsin the api route, how you could able to get the files byapi/documents/pdf/${document?.id} ?Oh, Snap, my bad. I'm terrible at replying on time... I'm hoping you've figured this out by this point, but to answer your question, it would be
api/${document?.id}... additionally, if you don't want to stream it in, you could provide a pre-signed URL (you would still need to make a call to get this data)Again my bad