目录

1.进程池项目回顾

2.不关闭文件描述符带来的危害

3.实验: 进程替换后的新程序读取泄漏的文件

4.解决"不关闭文件描述符带来的危害"的方法

5.进程池项目的改进方法3: 让子进程一开始就不继承得到的写端,使用O_CLOEXEC

方法3的完整代码

运行结果

1.进程池项目回顾

之前在OS47.【Linux】进程间通信 进程池项目的改进版文章说过这个进程池项目的问题:

父进程创建了子进程,子进程继承父进程打开的管道的写端,如果创建的子进程数量越多,那么最后一个子进程继承的父进程打开的管道的写端也就越多

2.不关闭文件描述符带来的危害

Thomas Zimmermann在File Descriptors During fork() and exec()文章是这样说的:

Unfortuantely, there is also one major drawback of this whole design. It’s too easy to leak file descriptors into a newly executed program. That’s what happened here.

The only standardized file descriptors are those for default input and output, and the one for error reporting. They are the file descriptors 0, 1 and 2 respectively. The newly executed program does not know about any other file descriptors opened by the process’ old program.

This is bad for two reasons. First of all it unnecessarily consumes resources. The maximum number of file descriptors per process is limited; typically to 1024. A program can run out of available file descriptors quickly if it doesn’t close file descriptors after their final use.

An even more sever problem is that the old program can leak information into the new program that the new program is not supposed to see. A file descriptor might refer to a file with sensitive data that only the original program was supposed to access. After the call to execl() the new program would be able to read this data as well.

由上得知,不关闭文件描述符带来的危害:

1.带来不必要的资源消耗,因为单个进程打开的文件描述符的数量是有限制的

        可以用ulimit -n命令查看:

2.安全问题: 泄漏的文件描述符会泄漏给新启动的程序

        如果旧程序的文件描述符指向被打开的、含有敏感数据的文件,而且该文件原则上只有旧程序才能访问,那么旧程序使用execl进程替换后,新启动的程序可通过文件描述符来获取敏感数据

       因为文件描述符指向的内容属于内核结构,用execl进程替换不会改变文件描述符指向的内容

下面做一个实验来演示

3.实验: 进程替换后的新程序读取泄漏的文件

创建一下文件:

test_leak_data/
├── makefile
├── new.cpp
├── old.cpp
└── sensitive_data.txt

makefile写入以下内容:

all: new.out old.out
new.out:new.cpp
	g++ -o $@ $^ -g -std=c++11
old.out:old.cpp
	g++ -o $@ $^ -g -std=c++11
.PHONY:clean
clean:
	rm -f new.out old.out

sensitive_data.txt写入以下内容:

这是敏感数据,原则上只有old.out才能访问,如果new.out访问到了将会泄漏数据!

old.cpp写入以下内容:

#include <unistd.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <iostream>
int main() 
{
    int sensitive_file_fd=open("./sensitive_data.txt",O_RDONLY);
    std::cout<<"旧进程: 已经打开了敏感文件,fd是3"<<std::endl;
    if (sensitive_file_fd == -1)
    {
        perror("open failed");
        return -1;
    }
    if (execl("./new.out", "new.out", (char *)NULL) == -1)
    {
        perror("execl failed");
        return -2;
    }
    return 0;
}

new.cpp写入以下内容:

#include <unistd.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <iostream>
int main() 
{
    std::cout<<"新进程: 尝试枚举所有的大于2的fd"<<std::endl;
    for (int fd=3;fd<10;fd++)
    {
        char buf[100];
        lseek(fd, 0, SEEK_SET);  // 重置文件指针到开头
        ssize_t n = read(fd, buf, sizeof(buf) - 1);
        if (n > 0) 
        {
            buf[n] = '\0';
            printf("已窃取敏感文件的文件描述符 %d 的内容: %s\n", fd, buf);       
        }
        else
           printf("新进程: 未窃取到文件描述符 %d 的内容\n", fd);
    }
    return 0;
}

运行结果:

​

4.解决"不关闭文件描述符带来的危害"的方法

为了避免不关闭文件描述符带来的危害,Thomas Zimmermann也给出了他的方法:

To prevent this, file descriptors have to be closed before the new program gets loaded by the kernel. Fortunately, there’s a simple way of ensuring this constraint: the O_CLOEXEC flag.

At the very beginning of this blog entry, we opened a file with the following code fragment.

int fd0 = open("/home/joe_user/my_file.txt", O_RDWR|O_CREAT, S_IRUSR|S_IWUSR|S_IRGRP);

By or-ing O_CLOEXEC into the flags argument, the returned file descriptor will be closed before the process, or any forked process, starts executing a new program.

int fd0 = open("/home/joe_user/my_file.txt", O_RDWR|O_CREAT|O_CLOEXEC, S_IRUSR|S_IWUSR|S_IRGRP);

How would our example look if we opened the file like this? During the fork the file descriptor and file-descriptor table are copied into the new process. The file descriptor carries the O_CLOEXEC flag in its entry in the file-descriptor table, but nothing else changed.

The big change comes when the new process executes execl(). Before loading the new program into the process, the kernel goes through the process’ file-descriptor table and closes all file descriptors that have the O_CLOEXEC flag set. Afterwards the resulting file data structures look like this.

​

The O_CLOEXEC flag can also be set and cleared with a call to fstat() fcntl()1 after the file descriptor has been created. But setting it later creates the possibility of meanwhile calling execl() and leaking file descriptors.

In practice, the safest strategy is to always set O_CLOEXEC when a new file descriptor gets created; and explictly clear the flag right before the call to execl() for those file descriptors that are supposed to be handed over to the new program.

使用O_CLOEXEC标志

修改上方实验的old.cpp中打开敏感文件的方式:

int sensitive_file_fd=open("./sensitive_data.txt",O_RDONLY|O_CLOEXEC);

运行结果: 新进程无法窃取机密文件内容

因为在进程替换前,操作系统会关闭带有O_CLOEXEC标志的文件描述符指向的文件

5.进程池项目的改进方法3: 让子进程一开始就不继承得到的写端,使用O_CLOEXEC

*注: 进程池项目的改进方法1和方法2的实现在OS47.【Linux】进程间通信 进程池项目的改进版文章

从Thomas Zimmermann文章提供的思路,虽然他的文章是以进程替换为新程序为例的,但是我认为O_CLOEXE应该也可以用于匿名管道的创建,让fork()后产生子进程不要继承父进程的管道的写端

确实我也找到了有关O_CLOEXE的资料,来自Linux内核的邮件:

https://lore.kernel.org/all/200705311809.l4VI9F9X009556@devserv.devel.redhat.com/

https://lore.kernel.org/all/200805062118.m46LI7AF004035@devserv.devel.redhat.com/

引入O_CLOEXEC的内核邮件.zip两个网址打包的邮件包下载:引入O_CLOEXEC的内核邮件.zip

下面摘录一下邮件中O_CLOEXEC被设计出来的原因:

https://lore.kernel.org/all/200705311809.l4VI9F9X009556@devserv.devel.redhat.com/

Ulrich Drepper <drepper@redhat.com>

Subject: [PATCH 00/18] flag parameters摘录: 

In some applications this can happen frequently.  Take a web browser.  One
thread opens a file and another thread starts, say, an external PDF viewer.
The result can even be a security issue if that open file descriptor refers
to a sensitive file and the external program can somehow be tricked into
using that descriptor. (这个观点和Thomas Zimmermann提到的一样,即安全问题)

Just adding O_CLOEXEC support to open() doesn't solve the whole set of
problems.  There are other ways to create file descriptors (socket,
epoll_create, Unix domain socket transfer, etc).  These can and should
be addressed separately though.  open() is such an easy case that it makes
not much sense putting the fix off.

在邮件线程的概览的最后,dean gaudet <dean@arctic.org>提出了关于pipe函数的建议

https://lore.kernel.org/all/Pine.LNX.4.64.0706091923580.10324@twinlark.arctic.org/

dean gaudet <dean@arctic.org>

Subject: Re: [PATCH] Introduce O_CLOEXEC (take >2)摘录:
nice.  i proposed something like this 8 or so years ago... the problem is 
that you've also got to deal with socket(2), socketpair(2), accept(2), 
pipe(2), dup(2), dup2(2), fcntl(F_DUPFD)... everything which creates new 
fds. (也需要处理这些会创建新的文件描述符的东西socket(2)、socketpair(2)、accept(2)、pipe(2)、dup(2)、dup2(2)、fcntl(F_DUPFD))

really what is desired is fork/clone with selective duping of fds.  i.e. 
you supply the list of what will become fd 0,1,2 in the child.

-dean

一年后,Ulrich Drepper上传了和pipe2函数有关的O_CLOEXEC补丁:

https://lore.kernel.org/all/200805062118.m46LI7AF004035@devserv.devel.redhat.com/

Ulrich Drepper <drepper@redhat.com>

Subject: [PATCH 00/18] flag parameters摘录:

I modified the headers in <linux/*> to define the new values by
include <linux/fcntl.h> (使用O_CLOEXEC要包含这个头文件,当然也可以 #include <fcntl.h>)

and then simply define the new constantsbased on the O_* constants.  That's the safest a nd least troublesome way.  Otherwise we would have to create arch-specific headers.  None of the modified headers should be used by userlevel code.  Therefore the namespace pollution is no issue. 

Some interfaces are not exported at userlevel with enough flexibility
to allow extending them.  For those we need new userlevel interface.
This is the case for:

 - paccept
 - epoll_create2
 - dup3
 - pipe2 (那么就可以用pipe2来创建有O_CLOEXEC标志的匿名管道了)
 - inotify_init1

那么可以将进程项目的pipe改成pipe2,因为pipe2第二个参数flags用于传标志位,可以写O_CLOEXEC

方法3的完整代码

#include <string>
#include <vector>
#include <iostream>
#include <sys/types.h>
#include <sys/wait.h>
#include <fcntl.h>
#include <unistd.h>
#include <ctime>
#include <limits.h>
#include "task.hpp"
const unsigned int max_process_num=5;
class channel
{
public:
    channel(int pipefd,pid_t childpid,const std::string& childname)
    :_pipefd(pipefd)
    ,_childpid(childpid)
    ,_childname(childname)
    {}
 
    int _pipefd;//管道的写端文件描述符
    pid_t _childpid;//子进程的pid
    std::string _childname;//子进程的名字
};
 
void send_cmd()
{
    srand((unsigned int)time(nullptr));
    //方法1: 随机选
    for (int i=0;i<10;i++)
    {
        sleep(1);
        //选择任务,这里使用随机数模拟
        int cmdcode = rand() % INT_MAX;
        
        //选择管道(等价于选择子进程)
        //fd的范围必须在[min_bound,max_bound]闭区间内
        //min_bound=4
        //max_bound=4+max_process_num-1
        int random = rand() % max_process_num; 
        int pipefd=4+random;
        //传输任务
        write(pipefd,&cmdcode,sizeof(int));
    }
}
 
void execute_cmd()
{
    for(;;)
    {
        sleep(1);
        int cmdcode=0;
        int n=read(0,&cmdcode,sizeof(int));
        if (n==sizeof(int))
        {
            std::cout<<getpid()<<"子进程: 执行"<<cmdcode<<"号任务"<<std::endl;
        }
        else if (n==0)
            break;
        else
        {
            std::cerr<<"非法的任务码! "<<std::endl;
        }
    }
}
 
void init_process_pool(std::vector<channel>& channels)
{
    std::vector<int> pipe_write_fd;
    for (int i=0;i<max_process_num;i++)
    {
        int pipefd[2];
        int pipe_ret=pipe2(pipefd,O_CLOEXEC);
         if (pipe_ret==-1)
             std::cerr<<"pipe创建管道失败! "<<std::endl;
 
        pid_t fork_ret=fork();       
        if (fork_ret==0)
        {
            //子进程只读管道,关闭写端
            close(pipefd[1]);
            for (auto elem:pipe_write_fd)
                close(elem);
            dup2(pipefd[0],0);
            execute_cmd();
            //执行完后关闭读端
            close(pipefd[0]);
            exit(0);
        }
        else if (fork_ret>0)
        {
            //父进程只写管道,关闭读端
            close(pipefd[0]);
            pipe_write_fd.push_back(pipefd[1]);
            channels.push_back(channel(pipefd[1],fork_ret,std::string("child process ")+std::to_string(i)));
        }
        else//fork_ret==-1
        {
            std::cerr<<"fork创建子进程失败! "<<std::endl;
        }
    }
    //send_cmd不能放循环里面,否则只会创建一个子进程
   
    send_cmd();
}
 
void print_channels(std::vector<channel>& channels)
{
    for (auto obj:channels)
    {
        std::cout<<"子进程名: "<<obj._childname<<", ";
        std::cout<<"PID: "<<obj._childpid<<", ";
        std::cout<<"管道写端描述符: "<<obj._pipefd<<std::endl;
    }
}
 
void clean_process_pool(std::vector<channel>& channels)
{
    for (auto& obj:channels)
    {
        close(obj._pipefd);
        waitpid(obj._childpid,nullptr,0);
        std::cout<<obj._childpid<<"子进程退出"<<std::endl;
    }
}

int main()
{
    std::vector<channel> channels;
    init_process_pool(channels);
    std::cout<<"所有子进程的文件描述符: "<<std::endl;
    for (auto& obj:channels)
    {
        //等价为ls -l /proc/pid/fd
        std::cout<<obj._childpid<<"子进程的文件描述符为: "<<std::endl;
        std::string str="ls -l /proc/"+std::to_string(obj._childpid)+"/fd";
        system(str.c_str());
    }
    clean_process_pool(channels);
    std::cout<<"父进程退出"<<std::endl;
    return 0;
}

运行结果