FastDFS + Nginx 反向代理缓存 安装与配置
作者：斯巴达克斯
本文来自 http://www.linux178.com/storage/fastdfs-nginx-cache.html
时间：March 26, 2015
很详细 我参照学习了
分类：存储
操作系统 CentOS release 6.5 (Final) 64
nginx相关软件
nginx-1.4.7 下载地址: http://nginx.org/en/download.html#nginx清除缓存模块 ngx_cache_purge-2.1 http://labs.frickle.com/nginx_ngx_cache_purge/pcre-8.36 ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.36.tar.gzzlib库 http://zlib.net/zlib-1.2.8.tar.gz
FastDFS相关软件
FastDFS_v5.04http://sourceforge.net/projects/fastdfs/files
FastDFS Nginx Module 1.16http://sourceforge.net/projects/fastdfs/files/FastDFS%20Nginx%20Module%20Source%20Code/
libfastcommonhttps://github.com/happyfish100/libfastcommon.git
拓扑图
二、tracker的安装与配置
1.安装
以下操作是在 tracker 10.10.10.124上操作
通过rz工具，把软件上传到/usr/local/src目录下，如果没有rz工具，可以使用 yum -y install lrzsz 安装
[root@tracker src]# lsFastDFS_v5.04.tar.gz
[root@tracker src]# tar xf FastDFS_v5.04.tar.gz
[root@tracker src]# cd FastDFS
[root@tracker FastDFS]# ./make.sh
执行make.sh 遇到的错误
看 源码目录下的INSTALL文件发现，需要安装 libfastcommon
####### the github address: https://github.com/happyfish100/libfastcommon.git ######
到如上的网址下载 libfastcommon进行安装，上传至服务器的src目录下
[root@tracker src]# unzip libfastcommon-master.zip
[root@tracker src]# cd libfastcommon-master
[root@tracker libfastcommon-master]# ls
HISTORY INSTALL libfastcommon.spec make.sh README src
[root@tracker libfastcommon-master]# ./make.sh
[root@tracker libfastcommon-master]# ./make.sh install
mkdir -p /usr/lib64install -m 755 libfastcommon.so /usr/lib64mkdir -p /usr/include/fastcommoninstall -m 644 common_define.h hash.h chain.h logger.h base64.h shared_func.h pthread_func.h ini_file_reader.h _os_bits.h sockopt.h sched_thread.h http_func.h md5.h local_ip_func.h avl_tree.h ioevent.h ioevent_loop.h fast_task_queue.h fast_timer.h process_ctrl.h fast_mblock.h connection_pool.h /usr/include/fastcommon
至此libfastcommon安装完成。
再回到 FastDFS目录下来安装FastDFS
[root@tracker libfastcommon-master]# cd ../FastDFS
[root@tracker FastDFS]# ./make.sh
[root@tracker FastDFS]# ./make.sh install
[root@tracker FastDFS]# ll
]# # ##### 所有的配置文件在 /etc/fdfs/ 目录下
[root@tracker FastDFS]# ll
[root@tracker FastDFS]#
至此 FastDFS在tracker上安装完成，以下进行配置
2. 配置
进入到 /etc/fdfs 目录下，复制模版文件到 tracker.conf
[root@tracker fdfs]# cp tracker.conf.sample tracker.conf
创建数据文件和日志文件目录
[root@tracker fdfs]# mkdir -pv /data/fastdfs/tracker
mkdir: 已创建目录 "/data"mkdir: 已创建目录 "/data/fastdfs"mkdir: 已创建目录 "/data/fastdfs/tracker"
编辑 tracker.conf 文件，测试的时候只需要修改以下参数即可
disabled=false
#启用配置文件port=22122
#设置 tracker 的端口号base_path=/data/fastdfs/tracker
#设置 tracker 的数据文件和日志目录（需预先创建）http.server_port=8080
#设置 http 端口号
这个http.server_port=8080 指的是在tracker服务器上启动http服务进程，如:apache或者nginx 启动时所监听的端口，这个似乎是可以不用管的，因为tracker本身就没有安装http服务
3.运行
直接使用 fdfs_trackerd 来启动tracker进程，然后使用netstat 查看端口是否起来。
[root@tracker fdfs]# fdfs_trackerd /etc/fdfs/tracker.conf restart
[root@tracker fdfs]# netstat -antp | grep trackerdtcp
0 0 0.0.0.0:22122 0.0.0.0:* LISTEN 14520/fdfs_trackerd [root@tracker fdfs]#
也可以查看日志看服务是否正常运行
[root@tracker tracker]# cat /data/fastdfs/tracker/logs/trackerd.log
4.设置开机自动启动
[root@tracker tracker]# echo "/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart" >> /etc/rc.local
[root@tracker tracker]# cat /etc/rc.local
#!/bin/sh
## This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.touch
/var/lock/subsys/local/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart
[root@tracker tracker]#
至此tracker上的安装与配置结束。
三、storage的安装与配置
1.安装 以下操作是在 另一台storage1上操作
利用上传 FastDFS_v5.04.tar.gz 和 libfastcommon-master.zip 至 storage服务器的/usr/local/src 目录下
[root@server1 src]# rz
rz waiting to receive.Starting zmodem transfer. Press Ctrl+C to cancel.Transferring FastDFS_v5.04.tar.gz... 100% 335 KB 335 KB/sec 00:00:01 0 Errors
[root@server1 src]# ls
FastDFS_v5.04.tar.gz
[root@server1 src]# rz
rz waiting to receive.Starting zmodem transfer. Press Ctrl+C to cancel.Transferring libfastcommon-master.zip... 100% 95 KB 95 KB/sec 00:00:01 0 Errors
[root@server1 src]# ll
总用量 432-rw-r--r-- 1 root root 343477 10月 27 09:26 FastDFS_v5.04.tar.gz-rw-r--r-- 1 root root 97548 10月 28 11:14 libfastcommon-master.zip
[root@server1 src]#
解压安装 libfastcommon
[root@server1 src]# unzip libfastcommon-master.zip
[root@server1 libfastcommon-master]# ll
[root@server1 libfastcommon-master]# ./make.sh
cc -Wall -D_FILE_OFFSET_BITS=64 -g -DDEBUG_FLAG -DOS_LINUX -DIOEVENT_USE_EPOLL -c -fPIC -o hash.lo hash.c cc -Wall -D_FILE_OFFSET_BITS=64 -g -DDEBUG_FLAG -DOS_LINUX -DIOEVENT_USE_EPOLL -c -fPIC -o chain.lo chain.c cc -Wall -D_FILE_OFFSET_BITS=64 -g -DDEBUG_FLAG -DOS_LINUX -DIOEVENT_USE_EPOLL -c -fPIC -o shared_func.lo shared_func.c cc -Wall -D_FILE_OFFSET_BITS=64 -g -DDEBUG_FLAG -DOS_LINUX -DIOEVENT_USE_EPOLL -c -fPIC -o ini_file_reader.lo ini_file_reader.c cc -Wall -D_FILE_OFFSET_BITS=64 -g -DDEBUG_FLAG -DOS_LINUX -DIOEVENT_USE_EPOLL -c -fPIC -o logger.lo logger.c cc -Wall -D_FILE_OFFSET_BITS=64 -g -DDEBUG_FLAG -DOS_LINUX -DIOEVENT_USE_EPOLL -c -fPIC -o sockopt.lo sockopt.c cc -Wall -D_FILE_OFFSET_BITS=64 -g -DDEBUG_FLAG -DOS_LINUX -DIOEVENT_USE_EPOLL -c -fPIC -o base64.lo base64.c ...
[root@server1 libfastcommon-master]# ./make.sh install
解压安装 FastDFS_v5.04.tar.gz
[root@server1 src]# tar xf FastDFS_v5.04.tar.gz [root@server1 src]# cd FastDFS[root@server1 FastDFS]# ll总用量 132drwxr-xr-x 3 8980 users 4096 9月 21 22:14 clientdrwxr-xr-x 2 8980 users 4096 9月 21 22:14 commondrwxr-xr-x 2 8980 users 4096 9月 21 22:14 conf-rw-r--r-- 1 8980 users 35067 9月 21 22:14 COPYING-3_0.txt-rw-r--r-- 1 8980 users 2802 9月 21 22:14 fastdfs.spec-rw-r--r-- 1 8980 users 31224 9月 21 22:14 HISTORYdrwxr-xr-x 2 8980 users 4096 9月 21 22:14 init.d-rw-r--r-- 1 8980 users 7755 9月 21 22:14 INSTALL-rwxr-xr-x 1 8980 users 5735 9月 21 22:14 make.shdrwxr-xr-x 2 8980 users 4096 9月 21 22:14 php_client-rw-r--r-- 1 8980 users 2380 9月 21 22:14 README.md-rwxr-xr-x 1 8980 users 1768 9月 21 22:14 restart.sh-rwxr-xr-x 1 8980 users 1680 9月 21 22:14 stop.shdrwxr-xr-x 4 8980 users 4096 9月 21 22:14 storagedrwxr-xr-x 2 8980 users 4096 9月 21 22:14 testdrwxr-xr-x 2 8980 users 4096 9月 21 22:14 tracker[root@server1 FastDFS]# [root@server1 FastDFS]# ./make.sh cc -Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O -DDEBUG_FLAG -DOS_LINUX -DIOEVENT_USE_EPOLL -c -o ../common/fdfs_global.o ../common/fdfs_global.c -I../common -I/usr/include/fastcommoncc -Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O -DDEBUG_FLAG -DOS_LINUX -DIOEVENT_USE_EPOLL -c -o tracker_proto.o tracker_proto.c -I../common -I/usr/include/fastcommoncc -Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O -DDEBUG_FLAG -DOS_LINUX -DIOEVENT_USE_EPOLL -c -o tracker_mem.o tracker_mem.c -I../common -I/usr/include/fastcommoncc -Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O -DDEBUG_FLAG -DOS_LINUX -DIOEVENT_USE_EPOLL -c -o tracker_service.o tracker_service.c -I../common -I/usr/include/fastcommoncc -Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O -DDEBUG_FLAG -DOS_LINUX -DIOEVENT_USE_EPOLL -c -o tracker_status.o tracker_status.c -I../common -I/usr/include/fastcommon...[root@server1 FastDFS]# ./make.sh installmkdir -p /usr/binmkdir -p /etc/fdfscp -f fdfs_trackerd /usr/binif [ ! -f /etc/fdfs/tracker.conf.sample ]; then cp -f ../conf/tracker.conf /etc/fdfs/tracker.conf.sample; fimkdir -p /usr/binmkdir -p /etc/fdfscp -f fdfs_storaged /usr/binif [ ! -f /etc/fdfs/storage.conf.sample ]; then cp -f ../conf/storage.conf /etc/fdfs/storage.conf.sample; fimkdir -p /usr/binmkdir -p /etc/fdfsmkdir -p /usr/lib64cp -f fdfs_monitor fdfs_test fdfs_test1 fdfs_crc32 fdfs_upload_file fdfs_download_file fdfs_delete_file fdfs_file_info fdfs_appender_test fdfs_appender_test1 fdfs_append_file fdfs_upload_appender /usr/binif [ 0 -eq 1 ]; then cp -f libfdfsclient.a /usr/lib64; fiif [ 1 -eq 1 ]; then cp -f libfdfsclient.so /usr/lib64; fimkdir -p /usr/include/fastdfscp -f ../common/fdfs_define.h ../common/fdfs_global.h ../common/mime_file_parser.h ../common/fdfs_http_shared.h ../tracker/tracker_types.h ../tracker/tracker_proto.h ../tracker/fdfs_shared_func.h ../storage/trunk_mgr/trunk_shared.h tracker_client.h storage_client.h storage_client1.h client_func.h client_global.h fdfs_client.h /usr/include/fastdfsif [ ! -f /etc/fdfs/client.conf.sample ]; then cp -f ../conf/client.conf /etc/fdfs/client.conf.sample; fi
安装完成后，可执行文件在/usr/bin/下以 fdfs 开头
[root@server1 FastDFS]# ll /usr/bin/fdfs_*-rwxr-xr-x 1 root root 257631 10月 28 14:18 /usr/bin/fdfs_appender_test-rwxr-xr-x 1 root root 257424 10月 28 14:18 /usr/bin/fdfs_appender_test1-rwxr-xr-x 1 root root 247664 10月 28 14:18 /usr/bin/fdfs_append_file-rwxr-xr-x 1 root root 247340 10月 28 14:18 /usr/bin/fdfs_crc32-rwxr-xr-x 1 root root 247723 10月 28 14:18 /usr/bin/fdfs_delete_file-rwxr-xr-x 1 root root 248850 10月 28 14:18 /usr/bin/fdfs_download_file-rwxr-xr-x 1 root root 248584 10月 28 14:18 /usr/bin/fdfs_file_info-rwxr-xr-x 1 root root 260848 10月 28 14:18 /usr/bin/fdfs_monitor-rwxr-xr-x 1 root root 886610 10月 28 14:18 /usr/bin/fdfs_storaged-rwxr-xr-x 1 root root 264031 10月 28 14:18 /usr/bin/fdfs_test-rwxr-xr-x 1 root root 263208 10月 28 14:18 /usr/bin/fdfs_test1-rwxr-xr-x 1 root root 374872 10月 28 14:18 /usr/bin/fdfs_trackerd-rwxr-xr-x 1 root root 248730 10月 28 14:18 /usr/bin/fdfs_upload_appender-rwxr-xr-x 1 root root 249676 10月 28 14:18 /usr/bin/fdfs_upload_file[root@server1 FastDFS]#
配置文件在 /etc/fdfs 目录下
[root@server1 FastDFS]# ll /etc/fdfs/总用量 20-rw-r--r-- 1 root root 1461 10月 28 14:18 client.conf.sample-rw-r--r-- 1 root root 7647 10月 28 14:18 storage.conf.sample-rw-r--r-- 1 root root 7102 10月 28 14:18 tracker.conf.sample[root@server1 FastDFS]#
2.配置
进入到 /etc/fdfs 目录下，复制/usr/local/src/FastDFS/conf/* 下面的所有文件到当前目录下(/etc/fdfs)
[root@server1 FastDFS]# cd /etc/fdfs/[root@server1 fdfs]# cp /usr/local/src/FastDFS/conf/* .[root@server1 fdfs]# ll总用量 64-rw-r--r-- 1 root root 1461 10月 29 14:33 client.conf-rw-r--r-- 1 root root 858 10月 29 14:33 http.conf-rw-r--r-- 1 root root 31172 10月 29 14:33 mime.types-rw-r--r-- 1 root root 3697 10月 29 14:34 mod_fastdfs.conf-rw-r--r-- 1 root root 7646 10月 28 14:34 storage.conf-rw-r--r-- 1 root root 105 10月 29 14:33 storage_ids.conf-rw-r--r-- 1 root root 7102 10月 29 14:33 tracker.conf[root@server1 fdfs]#
编辑配置文件 storage.conf 测试的时候，只需修改以下内容即可
[root@server1 fdfs]# vim storage.confdisabled=false #启用配置文件group_name=group1 #组名，根据实际情况修改port=23000 #设置 storage 的端口号base_path=/data/fastdfs/storage #设置 storage 的日志目录（需预先创建）store_path_count=1 #存储路径个数，需要和 store_path 个数匹配store_path0=/data/fastdfs/storage #存储路径tracker_server=10.10.10.124:22122 #tracker 服务器的 IP 地址和端口号http.server_port=8080 #设置storage上启动的http服务的端口号，如安装的nginx的端口号
3.运行
[root@server1 fdfs]# fdfs_storaged /etc/fdfs/storage.conf restart
查看端口是否起来
[root@server1 fdfs]# netstat -antp | grep storagetcp 0 0 0.0.0.0:23000 0.0.0.0:* LISTEN 10316/fdfs_storaged [root@server1 fdfs]#
观察日志 /data/fastdfs/storage/logs/storaged.log 看运行情况
mkdir data path: FB ...mkdir data path: FC ...mkdir data path: FD ...mkdir data path: FE ...mkdir data path: FF ...data path: /data/fastdfs/storage/data, mkdir sub dir done.[2014-10-28 14:42:55] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0[2014-10-28 14:42:55] INFO - file: storage_func.c, line: 175, tracker_client_ip: 10.10.10.120, my_server_id_str: 10.10.10.120, g_server_id_in_filename: 2013923850[2014-10-28 14:42:55] INFO - local_host_ip_count: 2, 127.0.0.1 10.10.10.120[2014-10-28 14:42:55] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 10.10.10.124:22122, as a tracker client, my ip is 10.10.10.120[2014-10-28 14:43:25] INFO - file: tracker_client_thread.c, line: 1235, tracker server 10.10.10.124:22122, set tracker leader: 10.10.10.124:22122
看到先创建二级目录，然后已经成功连接到tracker
可以使用 fdfs_monitor 来查看一下storage的状态，看是否已经成功注册到了tracker
[root@server2 logs]# fdfs_monitor /etc/fdfs/storage.conf
也可以以下命令来监控服务器的状态
[root@tracker ~]# fdfs_monitor /etc/fdfs/client.conf
结果如下:
[2014-10-28 15:57:54] DEBUG - base_path=/data/fastdfs/storage, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0server_count=1, server_index=0tracker server is 10.10.10.124:22122group count: 1Group 1:group name = group1disk total space = 14020 MBdisk free space = 10267 MBtrunk free space = 0 MBstorage server count = 2active server count = 2storage server port = 23000storage HTTP port = 8080store path count = 1subdir count per path = 256current write server index = 0current trunk file id = 0 Storage 1: id = 10.10.10.120 ip_addr = 10.10.10.120 ACTIVE http domain = version = 5.04 join time = 2014-10-28 14:42:43 up time = 2014-10-28 14:42:43 total storage = 14020 MB free storage = 10385 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8080 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 0 success_file_open_count = 0 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 0 success_file_write_count = 0 last_heart_beat_time = 2014-10-28 15:56:42 last_source_update = 1970-01-01 08:00:00 last_sync_update = 1970-01-01 08:00:00 last_synced_timestamp = 1970-01-01 08:00:00 Storage 2: id = 10.10.10.122 ip_addr = 10.10.10.122 (shizhouliang.mobimtech.local) ACTIVE http domain = version = 5.04 join time = 2014-10-28 15:53:51 up time = 2014-10-28 15:53:51 total storage = 14020 MB free storage = 10267 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8080 current_write_path = 0 source storage id = 10.10.10.120 if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 0 success_file_open_count = 0 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 0 success_file_write_count = 0 last_heart_beat_time = 2014-10-28 15:56:53 last_source_update = 1970-01-01 08:00:00 last_sync_update = 1970-01-01 08:00:00 last_synced_timestamp = 1970-01-01 08:00:00
看到ACTIVE,就说明已经成功注册到了tracker。
4.设置开机启动
[root@server1 fdfs]# echo "/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart" >> /etc/rc.local[root@server1 fdfs]# cat /etc/rc.local#!/bin/sh## This script will be executed *after* all the other init scripts.# You can put your own initialization stuff in here if you don't# want to do the full Sys V style init stuff.touch /var/lock/subsys/local/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart
至此 storage 已经成功安装，再把另外一台服务器也同样的步骤进行安装。
等第二台storage2(10.10.10.122)安装完成并成功启动后，看日志会发现
[2014-10-28 15:54:05] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 10.10.10.120:23000
说明已经成功连接到另外一台storage了。
也可以使用
[root@server2 logs]# fdfs_monitor /etc/fdfs/storage.conf
来查看一下storage2的状态
四、在storage上安装配置nginx
以下操作是在 storage1 10.10.10.120 上操作
1.安装nginx
上传 fastdfs-nginx-module_v1.16.tar.gz nginx-1.4.7.tar.gz pcre-8.36.tar.gz zlib-1.2.8.tar.gz 到storage服务器上的/usr/local/src 目录下
[root@server1 src]# rzrz waiting to receive.Starting zmodem transfer. Press Ctrl+C to cancel.Transferring fastdfs-nginx-module_v1.16.tar.gz... 100% 17 KB 17 KB/sec 00:00:01 0 Errors Transferring nginx-1.4.7.tar.gz... 100% 751 KB 751 KB/sec 00:00:01 0 Errors Transferring pcre-8.36.tar.gz... 100% 1962 KB 1962 KB/sec 00:00:01 0 Errors Transferring zlib-1.2.8.tar.gz... 100% 557 KB 557 KB/sec 00:00:01 0 Errors
解压各个文件，然后编译安装nginx
[root@server1 src]# tar xf nginx-1.4.7.tar.gz [root@server1 src]# tar xf pcre-8.36.tar.gz [root@server1 src]# tar xf zlib-1.2.8.tar.gz [root@server1 src]# tar xf fastdfs-nginx-module_v1.16.tar.gz [root@server1 src]# cd nginx-1.4.7[root@server1 nginx-1.4.7]# lsauto CHANGES CHANGES.ru conf configure contrib html LICENSE man README src[root@server1 nginx-1.4.7]# ./configure --prefix=/usr/local/nginx --add-module=/usr/local/src/fastdfs-nginx-module/src --with-pcre=/usr/local/src/pcre-8.36 --with-zlib=/usr/local/src/zlib-1.2.8
出现以下错误:
cc -c -pipe -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/local/include/fastdfs -I /usr/local/include/fastcommon/ -I /usr/local/src/pcre-8.36 -I /usr/local/src/zlib-1.2.8 -I objs -I src/http -I src/http/modules -I src/mail \ -o objs/addon/src/ngx_http_fastdfs_module.o \ /usr/local/src/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c在包含自 /usr/local/src/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c：6 的文件中:/usr/local/src/fastdfs-nginx-module/src/common.c:21:25: 错误：fdfs_define.h：没有那个文件或目录/usr/local/src/fastdfs-nginx-module/src/common.c:22:20: 错误：logger.h：没有那个文件或目录/usr/local/src/fastdfs-nginx-module/src/common.c:23:25: 错误：shared_func.h：没有那个文件或目录/usr/local/src/fastdfs-nginx-module/src/common.c:24:25: 错误：fdfs_global.h：没有那个文件或目录/usr/local/src/fastdfs-nginx-module/src/common.c:25:21: 错误：sockopt.h：没有那个文件或目录/usr/local/src/fastdfs-nginx-module/src/common.c:26:23: 错误：http_func.h：没有那个文件或目录/usr/local/src/fastdfs-nginx-module/src/common.c:27:30: 错误：fdfs_http_shared.h：没有那个文件或目录/usr/local/src/fastdfs-nginx-module/src/common.c:28:25: 错误：fdfs_client.h：没有那个文件或目录/usr/local/src/fastdfs-nginx-module/src/common.c:29:27: 错误：local_ip_func.h：没有那个文件或目录/usr/local/src/fastdfs-nginx-module/src/common.c:30:30: 错误：fdfs_shared_func.h：没有那个文件或目录/usr/local/src/fastdfs-nginx-module/src/common.c:31:26: 错误：trunk_shared.h：没有那个文件或目录
以上的这个错误是由于没找到 头文件导致的，找了找原因,在编译fastdf模块的时候会找这个地方/usr/local/include寻找fastdfs主程序的fastcommon和fastdfs这2个目录下的头文件，但是在编译安装的fastdfs主程序的时候，却把这2个目录放到了/usr/include下面了，于是复制这2个目录到/usr/local/include下
[root@server1 src]# cp -rf /usr/include/fast* /usr/local/include/[root@server1 src]# ls /usr/local/include/fastcommon fastdfs[root@server1 src]#
于是再次编译
[root@server1 src]# cd nginx-1.4.7[root@server1 nginx-1.4.7]# ./configure --prefix=/usr/local/nginx --add-module=/usr/local/src/fastdfs-nginx-module/src --with-pcre=/usr/local/src/pcre-8.36 --with-zlib=/usr/local/src/zlib-1.2.8[root@server1 nginx-1.4.7]# make && make install
这次终于编译安装通过。
2.准备启动脚本
编辑 /etc/init.d/nginx，如下内容:
#!/bin/bash# nginx Startup script for the Nginx HTTP Server# it is v.1.4.7 version.# chkconfig: - 85 15# description: Nginx is a high-performance web and proxy server.# It has a lot of features, but it's not for everyone.# processname: nginx # pidfile: /usr/local/nginx/logs/nginx.pid# config: /usr/local/nginx/conf/nginx.conf nginxd=/usr/local/nginx/sbin/nginxnginx_config=/usr/local/nginx/conf/nginx.confnginx_pid=/usr/local/nginx/logs/nginx.pidnginx_lock=/var/lock/subsys/nginxRETVAL=0prog="nginx" # Source function library.. /etc/rc.d/init.d/functions # Source networking configuration.. /etc/sysconfig/network # Check that networking is up.[ ${NETWORKING} = "no" ] && exit 0[ -x $nginxd ] || exit 0 # Start nginx daemons functions.start() { nginx_is_run=`ps -ef | egrep 'nginx:\s*(worker|master)\s*process' | wc -l` if [ ${nginx_is_run} -gt 0 ];then echo "nginx already running...." exit 1 fi echo -n $"Starting $prog: " daemon $nginxd -c ${nginx_config} RETVAL=$? echo [ $RETVAL = 0 ] && touch ${nginx_lock} return $RETVAL} # Stop nginx daemons functions.stop() { echo -n $"Stopping $prog: " killproc $nginxd RETVAL=$? echo [ $RETVAL = 0 ] && rm -f ${nginx_lock} ${nginx_pid}} # Reload nginx config filereload() { echo -n $"Reloading $prog: " #kill -HUP `cat ${nginx_pid}` killproc $nginxd -HUP RETVAL=$? echo} # See how we were called.case "$1" in start) start ;; stop) stop ;; reload) reload ;; restart) stop start ;; status) status $prog RETVAL=$? ;; *) echo $"Usage: $prog {start|stop|restart|reload|status|help}" exit 1esac
3.将 FastDFS 的 nginx 插件模块的配置文件
copy 到 FastDFS 配置文件目录
[root@server1 src]# cp fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/
4.修改nginx配置文件
① 修改nginx的监听端口
将 server 段中的 listen 端口号改为 8080：
[root@server1 nginx]# vim /usr/local/nginx/conf/nginx.conflisten 8080;
② 在 server 段中添加fastdfs的配置
这里的group[1-3] 可以匹配 group1,group2,group3，当然group也可以是其他的名字
location ~ /group[1-3]/M00 { root /data/fastdfs/storage/data; ngx_fastdfs_module; }
/data/fastdfs/storage/data 这个是指向 真正存储文件的地方，该目录下有这些个二级目录
[root@server1 src]# cd /data/fastdfs/storage/data/[root@server1 data]# ll总用量 1036drwxr-xr-x 258 root root 4096 10月 28 14:42 00drwxr-xr-x 258 root root 4096 10月 28 14:42 01drwxr-xr-x 258 root root 4096 10月 28 14:42 02drwxr-xr-x 258 root root 4096 10月 28 14:42 03drwxr-xr-x 258 root root 4096 10月 28 14:42 04drwxr-xr-x 258 root root 4096 10月 28 14:42 05drwxr-xr-x 258 root root 4096 10月 28 14:42 06drwxr-xr-x 258 root root 4096 10月 28 14:42 07drwxr-xr-x 258 root root 4096 10月 28 14:42 08drwxr-xr-x 258 root root 4096 10月 28 14:42 09drwxr-xr-x 258 root root 4096 10月 28 14:42 0Adrwxr-xr-x 258 root root 4096 10月 28 14:42 0Bdrwxr-xr-x 258 root root 4096 10月 28 14:42 0Cdrwxr-xr-x 258 root root 4096 10月 28 14:42 0Ddrwxr-xr-x 258 root root 4096 10月 28 14:42 0Edrwxr-xr-x 258 root root 4096 10月 28 14:42 0Fdrwxr-xr-x 258 root root 4096 10月 28 14:42 10
5.修改 fastdfs的nginx模块的配置文件 mod_fastdfs.conf
一般只需改动以下几个参数即可：
base_path=/data/fastdfs/storage #保存日志目录tracker_server=10.10.10.124:22122 #tracker 服务器的 IP 地址以及端口号storage_server_port=23000 #storage 服务器的端口号group_name=group1 #当前服务器的 group 名url_have_group_name = true #文件 url 中是否有 group 名store_path_count=1 #存储路径个数，需要和 store_path 个数匹配store_path0=/data/fastdfs/storage #存储路径http.need_find_content_type=true # 从文件 扩展 名查 找 文件 类型 （ nginx 时 为true）group_count = 1 #设置组的个数
然后在末尾添加分组信息，目前只有一个分组，就只写一个
[group1]group_name=group1storage_server_port=23000store_path_count=1store_path0=/data/fastdfs/storage
6.建立 M00 至存储目录的符号连接
[root@server1 fdfs]# ln -s /data/fastdfs/storage/data /data/fastdfs/storage/data/M00[root@server1 fdfs]# ll /data/fastdfs/storage/data/M00lrwxrwxrwx 1 root root 26 10月 29 11:22 /data/fastdfs/storage/data/M00 -> /data/fastdfs/storage/data
至此nginx和FastDFS 插件模块设置完成
7.启动nginx
[root@server1 nginx]# chmod u+x /etc/init.d/nginx[root@server1 nginx]# chkconfig --add nginx[root@server1 nginx]# chkconfig nginx on[root@server1 nginx]# service nginx start正在启动 nginx： [确定][root@server1 nginx]# service nginx statusnginx (pid 26500) 正在运行...
查看nginx的日志 错误日志logs/error.log 看是否有问题
[2014-10-29 14:35:29] INFO - local_host_ip_count: 2, 127.0.0.1 10.10.10.120[2014-10-29 14:35:29] INFO - fastdfs apache / nginx module v1.15, response_mode=proxy, base_path=/tmp, url_have_group_name=1, group_count=1, connect_timeout=2, network_timeout=30, tracker_server_count=1, if_alias_prefix=, local_host_ip_count=2, anti_steal_token=0, token_ttl=0s, anti_steal_secret_key length=0, token_check_fail content_type=, token_check_fail buff length=0, load_fdfs_parameters_from_tracker=1, storage_sync_file_max_delay=86400s, use_storage_id=0, storage server id count=0, flv_support=1, flv_extension=flv[2014-10-29 14:35:29] INFO - group 1. group_name=group1, storage_server_port=23000, path_count=1, store_path0=/data/fastdfs/storage
也可以访问 http://10.10.10.120:8080
看到没什么错误，至此在storage1上安装配置nginx结束。
再在storage2上以相同的步骤来安装nginx。
五、安装nginx反向代理及缓存服务器
以下操作在 10.10.10.123 上操作
1.安装nginx
将所需的软件包上传至服务器/usr/local/src下
[root@fastdfs-nginx ~]# cd /usr/local/src/[root@fastdfs-nginx src]# ls[root@fastdfs-nginx src]# rzrz waiting to receive.Starting zmodem transfer. Press Ctrl+C to cancel.Transferring nginx-1.4.7.tar.gz... 100% 751 KB 751 KB/sec 00:00:01 0 Errors Transferring ngx_cache_purge-2.1.tar.gz... 100% 10 KB 10 KB/sec 00:00:01 0 Errors Transferring pcre-8.36.tar.gz... 100% 1962 KB 1962 KB/sec 00:00:01 0 Errors Transferring zlib-1.2.8.tar.gz... 100% 557 KB 557 KB/sec 00:00:01 0 Errors
解压各个软件包
[root@fastdfs-nginx src]# tar xf nginx-1.4.7.tar.gz [root@fastdfs-nginx src]# tar xf ngx_cache_purge-2.1.tar.gz [root@fastdfs-nginx src]# tar xf pcre-8.36.tar.gz [root@fastdfs-nginx src]# tar xf zlib-1.2.8.tar.gz [root@fastdfs-nginx src]# cd nginx-1.4.7
编译安装nginx
[root@fastdfs-nginx nginx-1.4.7]# ./configure --prefix=/usr/local/nginx --add-module=/usr/local/src/ngx_cache_purge-2.1 --with-pcre=/usr/local/src/pcre-8.36/ --with-zlib=/usr/local/src/zlib-1.2.8checking for OS + Linux 2.6.32-431.el6.x86_64 x86_64checking for C compiler ... found + using GNU C compiler + gcc version: 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) checking for gcc -pipe switch ... foundchecking for gcc builtin atomic operations ... foundchecking for C99 variadic macros ... foundchecking for gcc variadic macros ... foundchecking for unistd.h ... foundchecking for inttypes.h ... found...Configuration summary + using PCRE library: /usr/local/src/pcre-8.36/ + OpenSSL library is not used + md5: using system crypto library + sha1: using system crypto library + using zlib library: /usr/local/src/zlib-1.2.8 nginx path prefix: "/usr/local/nginx" nginx binary file: "/usr/local/nginx/sbin/nginx" nginx configuration prefix: "/usr/local/nginx/conf" nginx configuration file: "/usr/local/nginx/conf/nginx.conf" nginx pid file: "/usr/local/nginx/logs/nginx.pid" nginx error log file: "/usr/local/nginx/logs/error.log" nginx http access log file: "/usr/local/nginx/logs/access.log" nginx http client request body temporary files: "client_body_temp" nginx http proxy temporary files: "proxy_temp" nginx http fastcgi temporary files: "fastcgi_temp" nginx http uwsgi temporary files: "uwsgi_temp" nginx http scgi temporary files: "scgi_temp"[root@fastdfs-nginx nginx-1.4.7]# make && make install
至此 nginx 以及 nginx cache purge 插件模块安装完成。
2.配置
准备nginx的启动脚本，内容同上面的storage安装时的脚本，并加入到系统的服务，开机启动
创建nginx的缓存目录
[root@fastdfs-nginx nginx]# mkdir -pv /var/cache/nginx/proxy_cache/tmpmkdir: 已创建目录 "/var/cache/nginx"mkdir: 已创建目录 "/var/cache/nginx/proxy_cache"mkdir: 已创建目录 "/var/cache/nginx/proxy_cache/tmp"
增加系统打开文件数的限制
[root@fastdfs-nginx nginx]# ulimit -SHn 102400[root@fastdfs-nginx nginx]# echo "ulimit -SHn 102400" >> /etc/rc.local
编辑 vim /usr/local/nginx/conf/nginx.conf 文件修改以下内容:
user nobody;worker_processes 1;error_log logs/error.log info;pid logs/nginx.pid;events { worker_connections 65535; use epoll;}http { include mime.types; default_type application/octet-stream; sendfile on; tcp_nopush on; keepalive_timeout 65; server_names_hash_bucket_size 128; client_header_buffer_size 32k; large_client_header_buffers 4 32k; client_max_body_size 300m; proxy_redirect off; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 90; proxy_send_timeout 90; proxy_read_timeout 90; proxy_buffer_size 16k; proxy_buffers 4 64k; proxy_busy_buffers_size 128k; proxy_temp_file_write_size 128k; #设置缓存存储路径、存储方式、分配内存大小、磁盘最大空间、缓存期限 proxy_cache_path /var/cache/nginx/proxy_cache levels=1:2 keys_zone=http-cache:500m max_size=10g inactive=30d; proxy_temp_path /var/cache/nginx/proxy_cache/tmp; #设置 group1 的服务器 upstream fdfs_group1 { server 10.10.10.120:8080 weight=1 max_fails=2 fail_timeout=30s; server 10.10.10.122:8080 weight=1 max_fails=2 fail_timeout=30s; } log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; server { listen 80; server_name localhost; access_log logs/host.access.log main; #设置 group1 的反向代理参数 location /group1/M00 { proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_cache http-cache; proxy_cache_valid 200 304 12h; proxy_cache_key $uri$is_args$args; proxy_pass http://fdfs_group1; expires 30d; } #设置清除缓存的访问权限 location ~ /purge(/.*) { allow 127.0.0.1; allow 10.10.10.0/24; deny all; proxy_cache_purge http-cache $1$is_args$args; } }}
使用 /usr/local/nginx/sbin/nginx -t 来测试配置文件语法的正确性
[root@fastdfs-nginx nginx]# ./sbin/nginx -tnginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is oknginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
3.启动nginx
[root@fastdfs-nginx nginx]# service nginx start正在启动 nginx： [确定]
查看监听的端口
[root@fastdfs-nginx conf]# netstat -antp | grep nginxtcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 20526/nginx [root@fastdfs-nginx conf]#
六、测试上传文件
以下操作在tracker服务器（10.10.10.124）上进行
1.创建client 使用的日志目录
[root@tracker tracker]# mkdir -pv /data/fastdfs/clientmkdir: 已创建目录 "/data/fastdfs/client"
2. 修改客户端的配置文件/etc/fdfs/client.conf
修改以下参数即可：
base_path=/data/fastdfs/client #日志存放路径tracker_server=10.10.10.124:22122 #tracker 服务器 IP 地址和端口号http.tracker_server_port=8080 #tracker 服务器的 http 端口号 ，这一项感觉没什么用处，tracker都没有启用http服务
3.上传一个图片
[root@tracker ~]# fdfs_upload_file /etc/fdfs/client.conf 1.jpggroup1/M00/00/00/CgoKeFRQqaaASsgNAAEgRqu1UiU712.jpg[root@tracker ~]#
得到了一个如上的一个地址 group1/M00/00/00/CgoKeFRQqaaASsgNAAEgRqu1UiU712.jpg
可以使用fdfs_file_info 来查看上传文件的信息
[root@tracker ~]# fdfs_file_info /etc/fdfs/client.conf group1/M00/00/00/CgoKeFRQqaaASsgNAAEgRqu1UiU712.jpgsource storage id: 0source ip address: 10.10.10.120file create timestamp: 2014-10-29 16:47:34file size: 73798file crc32: 2880786981 (0xABB55225)
这个信息里面可以看到 这个文件上传的时候 首先上传到了10.10.10.120这台服务器，文件的创建时间、文件的大小
4.使用浏览器访问一下该地址
完整的URL是这样的http://10.10.10.123/group1/M00/00/00/CgoKeFRQqaaASsgNAAEgRqu1UiU712.jpg
10.10.10.123 就是nginx反向代理缓存服务器
成功访问到图片
查看一下nginx的访问日志，可以看到状态码是 200OK.
[root@fastdfs-nginx ~]# tail -1 /usr/local/nginx/logs/host.access.log 10.10.10.46 - - [29/Oct/2014:16:48:43 +0800] "GET /group1/M00/00/00/CgoKeFRQqaaASsgNAAEgRqu1UiU712.jpg HTTP/1.1" 200 73798 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36" "-"[root@fastdfs-nginx ~]#
查看nginx的缓存目录，看是否已经缓存成功
[root@fastdfs-nginx ~]# ll -R /var/cache/nginx/proxy_cache/ /var/cache/nginx/proxy_cache/:总用量 8drwx------. 3 nobody nobody 4096 10月 29 16:47 edrwxr-xr-x. 2 nobody root 4096 10月 29 16:47 tmp/var/cache/nginx/proxy_cache/e:总用量 4drwx------. 2 nobody nobody 4096 10月 29 16:47 a8/var/cache/nginx/proxy_cache/e/a8:总用量 76-rw-------. 1 nobody nobody 74110 10月 29 16:47 54408edb44f74ab4da965506f54a6a8e 在此已经缓存成功/var/cache/nginx/proxy_cache/tmp:总用量 0[root@fastdfs-ngi
编辑回复