Java - 从文件压缩聊一聊I/O一二事

背景

有个文件压缩的需求，小伙伴一顿操作猛如虎，小文件那是咔咔一顿骚

可是突然一个几十兆的文件，跑了100秒还没出来。。。。

代码语言：javascript复制

/**
 * @author 小工匠
 * @version 1.0
 * @description: TODO
 * @date 2021/2/3 16:40
 * @mark: show me the code , change the world
 */
public class FileCompress {


    //要压缩的文件所在所存放位置
    public static String COMPRESS_FILE_PATH = "D:/test/1.pdf";

    //zip压缩包所存放的位置
    public static String ZIP_FILE = "D:/test/1.zip";

    //要压缩的文件
    public static File COMPRESS_FILE = null;

    //文件大小
    public static long FILE_SIZE = 0;

    //文件名
    public static String FILE_NAME = "";

    //文件后缀名
    public static String SUFFIX_FILE = "";

    static {

        File file = new File(COMPRESS_FILE_PATH);

        COMPRESS_FILE = file;

        FILE_NAME = file.getName();

        FILE_SIZE = file.length();

        SUFFIX_FILE = FILE_NAME.substring(FILE_NAME.indexOf('.'));
    }

   public static void main(String[] args) throws RunnerException {
			...........
			...........
			...........
	}
}

问题复现

为了说明问题，模拟下事发现场

Version1: no buffer

代码语言：javascript复制

 public static void zipFileVersion1() {
        File zipFile = new File(ZIP_FILE);
        try (ZipOutputStream zipOut = new ZipOutputStream(new FileOutputStream(zipFile))) {
            //开始时间
            long beginTime = System.currentTimeMillis();

            try (InputStream input = new FileInputStream(COMPRESS_FILE)) {
                zipOut.putNextEntry(new ZipEntry(FILE_NAME   1));
                int temp = 0;
                while ((temp = input.read()) != -1) {
                    zipOut.write(temp);
                }
            }
            long cost = (System.currentTimeMillis() - beginTime);

            System.out.println("fileSize:"   FILE_SIZE / 1024 / 1024    "M");
            System.out.println("zip file cost time:"   cost / 1000   "s");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

就压缩一个pdf ， 60来兆

问题很明显，连缓冲也不用，面子不能给呀

Version 2 : with buffer

代码语言：javascript复制

public static void zipFileVersion() {
        long beginTime = System.currentTimeMillis();
        FileOutputStream fos = null;
        ZipOutputStream zos = null;
        try {
            byte[] buffer = new byte[1024];

            fos = new FileOutputStream(ZIP_FILE);

            zos = new ZipOutputStream(fos);
            File srcFile = COMPRESS_FILE;
            FileInputStream fis = new FileInputStream(srcFile);
            zos.putNextEntry(new ZipEntry("artisan"   SUFFIX_FILE));
            int length;
            while ((length = fis.read(buffer)) > 0) {
                zos.write(buffer, 0, length);
            }
            zos.closeEntry();
            fis.close();
        } catch (IOException e) {
            System.out.println("Error : "   e);
        } finally {
            try {
                zos.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
            long cost = (System.currentTimeMillis() - beginTime);
            System.out.println("fileSize:"   FILE_SIZE / 1024 / 1024   "M");
            System.out.println("test zip file cost time:"   cost    "ms");
        }
    }

代码语言：javascript复制

public static void zipFileVersion2() { 
    	long beginTime = System.currentTimeMillis();
        File zipFile = new File(ZIP_FILE);
        try {
            ZipOutputStream zipOut = new ZipOutputStream(new BufferedOutputStream(new FileOutputStream(zipFile)));
            BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(zipOut);
            BufferedInputStream bufferedInputStream = new BufferedInputStream(new FileInputStream(COMPRESS_FILE));
            zipOut.putNextEntry(new ZipEntry("artisan"  SUFFIX_FILE));
            int temp = 0;
            while ((temp = bufferedInputStream.read()) != -1) {
                bufferedOutputStream.write(temp);
            }
            zipOut.closeEntry();
            bufferedInputStream.close();
            zipOut.close();
            long cost = (System.currentTimeMillis() - beginTime);

            System.out.println("fileSize:"   FILE_SIZE / 1024 / 1024   "M");
            System.out.println("zip file cost time:"   cost    "ms");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

wtf , 直接干到2.8秒，发生了什么

提速原因源码分析

我们先看一下，version1的核心读取文件的方法

FileInputStream#read

可以看到read0() 一个调用本地方法与原生操作系统进行交互，从磁盘中读取数据。每读取一个字节的数据就调用一次本地方法与操作系统交互，一个63M的文档，转换成直接，那得交互多少次…那耗时…

而如果使用缓冲区的话（这里假设初始的缓冲区大小足够放下63M的数据）那么只需要调用一次就行。因为缓冲区在第一次调用read()方法的时候会直接从磁盘中将数据直接读取到内存中，随后再一个字节一个字节的慢慢返回。

可以看到 BufferedInputStream内部封装了一个byte数组用于存放数据，默认大小是8192

Version 3 : nio - Channel

满足了吗？

上面都是传统I/O操作，不想用用nio么？

NIO中的Channel和ByteBuffer，它们的结构更加符合操作系统执行I/O的方式，所以其速度相比较于传统IO而言速度有了显著的提高。

Channel管道比作成铁路，buffer缓冲区比作成火车(运载着货物) .

NIO就是通过Channel管道运输着存储数据的Buffer缓冲区的来实现数据的处理

在NIO中能够产生FileChannel的有三个类分别是

FileInputStream
FileOutputStream
既能读又能写的RandomAccessFile

代码语言：javascript复制

 public static void zipFileVersion3() { 
        long beginTime = System.currentTimeMillis();
        File zipFile = new File(ZIP_FILE);
        try (ZipOutputStream zipOut = new ZipOutputStream(new FileOutputStream(zipFile));
             WritableByteChannel writableByteChannel = Channels.newChannel(zipOut)) {
            try (FileChannel fileChannel = new FileInputStream(COMPRESS_FILE).getChannel()) {
                zipOut.putNextEntry(new ZipEntry("artisan"   SUFFIX_FILE));
                fileChannel.transferTo(0, FILE_SIZE, writableByteChannel);
            }

            long cost = (System.currentTimeMillis() - beginTime);

            System.out.println("fileSize:"   FILE_SIZE / 1024 / 1024   "M");
            System.out.println("zip file cost time:"   cost    "ms");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

可以看到这里并没有使用ByteBuffer进行数据传输，而是使用了transferTo的方法。这个方法是将两个通道进行直连。

来看一下官方的说明

代码语言：javascript复制

This method is potentially much more efficient than a simple loop
* that reads from this channel and writes to the target channel.  Many
* operating systems can transfer bytes directly from the filesystem cache
* to the target channel without actually copying them.

大概意思就是使用transferTo的效率比循环一个Channel读取出来然后再循环写入另一个Channel好。操作系统能够直接传输字节从文件系统缓存到目标的Channel中，而不需要实际的copy阶段。

那什么是copy阶段呢？【从内核空间转到用户空间的一个过程】

Version 4 : nio - Channel With Buffer

代码语言：javascript复制

 public static void zipFileChannelBuffer() { 
        long beginTime = System.currentTimeMillis();
        File zipFile = new File(ZIP_FILE);
        try (ZipOutputStream zipOut = new ZipOutputStream(new BufferedOutputStream(new FileOutputStream(zipFile)));
             WritableByteChannel writableByteChannel = Channels.newChannel(zipOut)) {
            try (FileChannel fileChannel = new FileInputStream(COMPRESS_FILE).getChannel()) {
                zipOut.putNextEntry(new ZipEntry("artisan"   SUFFIX_FILE));
                fileChannel.transferTo(0, FILE_SIZE, writableByteChannel);
            }
            printInfo(beginTime);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

Version 5 : MMAP

NIO中新出的另一个特性就是内存映射文件，内存映射文件为什么速度快呢？其实是在内存中开辟了一段直接缓冲区,与数据直接作交互。

代码语言：javascript复制

public static void zipFileMMAP() {
        //开始时间
        long beginTime = System.currentTimeMillis();
        File zipFile = new File(ZIP_FILE);
        try (ZipOutputStream zipOut = new ZipOutputStream(new FileOutputStream(zipFile));
             WritableByteChannel writableByteChannel = Channels.newChannel(zipOut)) {
            zipOut.putNextEntry(new ZipEntry("artisan"   SUFFIX_FILE));

            //内存中的映射文件
            MappedByteBuffer mappedByteBuffer = new RandomAccessFile(COMPRESS_FILE_PATH, "r").getChannel()
                    .map(FileChannel.MapMode.READ_ONLY, 0, FILE_SIZE);

            writableByteChannel.write(mappedByteBuffer);
            long cost = (System.currentTimeMillis() - beginTime);

            System.out.println("fileSize:"   FILE_SIZE / 1024 / 1024   "M");
            System.out.println("mmap file cost time:"   cost    "ms");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

Version 6 : PIPE

Java NIO 管道是2个线程之间的单向数据连接。Pipe有一个source通道和一个sink通道。其中source通道用于读取数据，sink通道用于写入数据。

代码语言：javascript复制

 Whether or not a thread writing bytes to a pipe will block until another
 thread reads those bytes

大概意思就是写入线程会阻塞至有读线程从通道中读取数据。如果没有数据可读，读线程也会阻塞至写线程写入数据。直至通道关闭。

代码语言：javascript复制

 public static void zipFilePip() {

        long beginTime = System.currentTimeMillis();
        try(WritableByteChannel out = Channels.newChannel(new FileOutputStream(ZIP_FILE))) {
            Pipe pipe = Pipe.open();
            //异步任务
            CompletableFuture.runAsync(()->runTask(pipe));

            //获取读通道
            ReadableByteChannel readableByteChannel = pipe.source();
            ByteBuffer buffer = ByteBuffer.allocate(((int) FILE_SIZE)*10);
            while (readableByteChannel.read(buffer)>= 0) {
                buffer.flip();
                out.write(buffer);
                buffer.clear();
            }
        }catch (Exception e){
            e.printStackTrace();
        }
        printInfo(beginTime);

    }

    //异步任务
    public static void runTask(Pipe pipe) {

        try(ZipOutputStream zos = new ZipOutputStream(Channels.newOutputStream(pipe.sink()));
            WritableByteChannel out = Channels.newChannel(zos)) {
            System.out.println("Begin");
                zos.putNextEntry(new ZipEntry("artisan" SUFFIX_FILE));

                FileChannel jpgChannel = new FileInputStream(new File(COMPRESS_FILE_PATH)).getChannel();

                jpgChannel.transferTo(0, FILE_SIZE, out);

                jpgChannel.close();
        }catch (Exception e){
            e.printStackTrace();
        }
    }