当前位置: 首页 > news >正文

java byte 解压_java-如何使用Zstd-jni和字节缓冲区解压缩大...

我尝试使用ByteBuffers和Channels并行下载大量40 MB文件时解压缩.与使用Streams相比,使用Channels可以获得更高的吞吐量,我们需要这是一个非常高的吞吐量的系统,因为我们每天需要处理40 TB的文件,而这一部分目前是瓶颈.文件使用zstd-jni压缩.Zstd-jni具有用于解压缩字节缓冲区的api,但是使用它们时出现错误.如何使用zstd-jni一次解压缩字节缓冲区?

我在他们的测试中找到了这些示例,但是除非丢失了某些内容,否则使用ByteBuffers的示例似乎假定整个输入文件都适合一个ByteBuffer:

https://github.com/luben/zstd-jni/blob/master/src/test/scala/Zstd.scala

以下是我用于压缩和解压缩文件的代码.压缩代码效果很好,但是解压缩代码失败,错误为-70.

public static long compressFile(String inFile, String outFolder, ByteBuffer inBuffer, ByteBuffer compressedBuffer, int compressionLevel) throws IOException {

File file = new File(inFile);

File outFile = new File(outFolder, file.getName() + ".zs");

long numBytes = 0l;

try (RandomAccessFile inRaFile = new RandomAccessFile(file, "r");

RandomAccessFile outRaFile = new RandomAccessFile(outFile, "rw");

FileChannel inChannel = inRaFile.getChannel();

FileChannel outChannel = outRaFile.getChannel()) {

inBuffer.clear();

while(inChannel.read(inBuffer) > 0) {

inBuffer.flip();

compressedBuffer.clear();

long compressedSize = Zstd.compressDirectByteBuffer(compressedBuffer, 0, compressedBuffer.capacity(), inBuffer, 0, inBuffer.limit(), compressionLevel);

numBytes+=compressedSize;

compressedBuffer.position((int)compressedSize);

compressedBuffer.flip();

outChannel.write(compressedBuffer);

inBuffer.clear();

}

}

return numBytes;

}

public static long decompressFile(String originalFilePath, String inFolder, ByteBuffer inBuffer, ByteBuffer decompressedBuffer) throws IOException {

File outFile = new File(originalFilePath);

File inFile = new File(inFolder, outFile.getName() + ".zs");

outFile = new File(inFolder, outFile.getName());

long numBytes = 0l;

try (RandomAccessFile inRaFile = new RandomAccessFile(inFile, "r");

RandomAccessFile outRaFile = new RandomAccessFile(outFile, "rw");

FileChannel inChannel = inRaFile.getChannel();

FileChannel outChannel = outRaFile.getChannel()) {

inBuffer.clear();

while(inChannel.read(inBuffer) > 0) {

inBuffer.flip();

decompressedBuffer.clear();

long compressedSize = Zstd.decompressDirectByteBuffer(decompressedBuffer, 0, decompressedBuffer.capacity(), inBuffer, 0, inBuffer.limit());

System.out.println(Zstd.isError(compressedSize) + " " + compressedSize);

numBytes+=compressedSize;

decompressedBuffer.position((int)compressedSize);

decompressedBuffer.flip();

outChannel.write(decompressedBuffer);

inBuffer.clear();

}

}

return numBytes;

}

相关文章:

  • 查看Java项目中各类加载顺序_Web.xml在JAVA项目中的加载顺序是什么
  • vs2017nuget安装java包_VSCode使用dotnet命令安装Nuget包及配置Nuget源的方法
  • java dispatcher详解_SpringBoot之DispatcherServlet详解及源码解析
  • php注册程序,简单的php 用户注册程序[全套]
  • php unset 变量,php unset全局变量运用问题的深入解析
  • php chroot,PHP-FPM的chroot和chdir目录
  • php图片上传怎么跨域,php后端控制可跨域的域名并且允许图片跨域上传的方法介绍...
  • java 并发 关键字,java – 关于并发的最终关键字保证究竟是什么?
  • php foot文件连接,ThinkPHP5——引入公共部分head和foot(多种方法)
  • matlab 滤波器阶数生成,!!!!滤波器阶数问题!!!!
  • 文件上传显示进度条 php,文件上传进度条php代码
  • dw main.less php,less基础详解 - Python测试和开发 - OSCHINA - 中文开源技术交流社区
  • erlang mnesia php,erlang mnesia数据库事物应用
  • mysql5.5 删除表,MYSQL 5.5删除主键
  • matlab降低函数的峰谷差,基于matlab的简单的寻找波峰波谷处理方法