For one of the projects we have been working on, we had to be able to upload videos to the server, convert them to formats supported by browsers with <video> tag and serve them to the user. Since Ruby and Ruby on Rails are our tools of choice we used CarrierWave for file uploading and storage. Since Carrier Wave supports processing of input files into various versions, we decided to give it a try.

Escaping CarrierWave versions

CarrierWave Gem

Since processing video files takes a lot of time, we had to do it in the background so we used carrierwave-backgrounder. When looking for a way to do video processing, we found the gem carrierwave-video that used streamio-ffmpeg for transcoding and it seemed fine for our task.

Our Gemfile looked like this:

gem 'carrierwave', '0.10.0' 
gem 'carrierwave-backgrounder', '0.4.2'
gem 'carrierwave-video', '0.5.6'

To encode videos to proper formats we used encode_video method from carrierwave-video:

version :mp4 do 
process encode_video: [:mp4, { progress: :on_progress }]
end
version :webm do 
process encode_video: [:webm, { progress: :on_progress }]
end

After some time there was a need in the project to be able to cut out small pieces of videos and serve them independently.

  • At first we tried to do it the same way as previous processing:
version :mp4 do 
process encode: [:mp4, { progress: :on_progress }]
end
version :webm do 
process encode: [:webm, { progress: :on_progress }]
end
def encode(format, opts={}) 
encode_video(format, opts) do |movie, params|
params[:custom] = "-ss #{model.playback.start_in_hms} -t #{model.playback.duration_in_hms}"
end
end

Unfortunately this had a very serious drawback — since all of the versions were being processed from one file inputted to the uploader, they had to be transcoded every time. That was far from perfect as it took additional time and unnecessarily used resources so we hacked our way a bit: since the original video was already in the formats we needed, we could just pass those versions instead and copy the audio and video streams.

The Solution

To make the code less cluttered, we sprinkled it with a bit of DSL:

support_formats custom: proc { |model| "-ss #{model.start_time} -t #{model.duration}" }, 
source: proc { |model, version| model.parent.file.versions[version].file },
mode: :copy
def support_formats(support_opts={}) 
FORMATS.each do |version_name, opts|
opts = opts.reverse_merge(support_opts)
 (conditional = opts.delete(:if)) && (conditional = conditional.to_sym)
uploader = version(version_name, if: conditional) { process encode_format: [opts] }
uploader[:uploader].class_eval <<-RUBY, __FILE__, __LINE__ + 1
def full_filename(file)
file + ".#{version_name}.#{opts[:extension]}" # forcing the extension, otherwise ffmpeg got confused
end
RUBY
end
end
def encode_format(opts={}) 
cache_stored_file! if !cached?
 if opts[:mode] == :copy
opts[:video_codec] = 'copy'
opts[:audio_codec] = 'copy'
end
 opts[:custom] = opts[:custom].call(model) if opts[:custom].respond_to?(:call)
 source = source.call(model, version_name) if source.respond_to?(:call)
source = file if source.nil?
source = source.path if source.respond_to?(:path)
 # etc
end

Sign up for Untitled Kingdom newsletter

Fresh startup & app development infobites delivered to your inbox.

➡️Check our Privacy Policy⬅️

This setup kind of worked, but it posed a lot of problems: we didn’t have control over what versions were transcoded and we had to recreate each version if any of the transcoding failed. More than that, if any transcoding happened during deployment of new version, sidekiq had to be killed and restarted and it didn’t have a way of going back from where it started so the whole processing had to be either redone or ditched altogether and marked as crashed.

We have tried various solutions of mitigating this problem but unfortunately using carrierwave-backgrounder made everything more messy. That gem was great for simpler logic but unfortunately choked a bit when we tried extending it more. This also caused the logic of processing to be divided between non-logical parts, like processing code ending up partially in sidekiq worker (because it was easy to set custom worker when mounting an uploader), non-obvious or custom callbacks being thrown all over the place or processing starting non-asynchronously if we weren’t careful enough. The API got brittle and the whole codebase gradually became a mess.

class StreamUploadWorker < ::CarrierWave::Workers::StoreAsset
 def perform(*args)
set_args(*args) if args.present?
video = constantized_resource.find id
 # custom callbacks in model
run_callback video, :before_upload
super(*args)
video.reload
 run_callback video, :after_upload_success
 rescue
run_callback video, :after_upload_failure
video.broken! if video.respond_to?(:broken!)
 # logging
end
 def run_callback(video, callback)
video.send(callback) if video.respond_to?(callback)
end
end

The last straw…

The last straw, however, came with a new requirement. The project has matured enough to generate significant traffic and serving multimedia content from a dedicated server no longer seemed like a viable solution. We needed to have files on both a local server (for processing) and on some kind of cloud solution (for hosting). As CarrierWave versions are nothing more than differently named files in a directory, using them seemed like a bad idea considering the amount of patchwork needed. It was time to clean the house.

We solved the problem by ditching versions altogether. We created a separate model for storing files that could either have local or remote (fog) uploader attached. Then we wrote our own transcoding logic, top down, easy to understand, using streamio-ffmpeg directly with specified file as the source and putting it in the path CarrierWave expected.

class FileObject < ActiveRecord::Base 
belongs_to :owner, polymorphic: true
 class Local < FileObject
mount_uploader :file, FileObjectUploader
end
 class Cloud < FileObject
mount_uploader :file, AwsUploader
end
end
class Processor 
# partial, example code
 def recreate_versions!(video, file)
formats_with_options(video, file).each do |format, opts|
if video.parent.present?
original = video.parent.version(format)
file = original.file if original.respond_to?(:file)
end
 file_object = FileObject::Local.create!(owner: video, version: format)
filename = "#{SecureRandom.uuid}.#{opts[:extension]}"
 FileUtils.mkdir_p file_object.file.store_dir
destination = "#{file_object.file.store_dir}/#{filename}"
 transcode!(source, destination, format, opts)
file_object.update_column(:file, filename)
 S3UploadWorker.perform_async(file_object.id) unless Rails.env.development?
end
end
end

Summary

CarrierWave is a great solution for file uploading. It’s also a great solution for simple processing. The moral of the story is, you have to use the appropriate tools. Carrier Wave versions are not enough for complex processing or any processing that doesn’t use the original uploaded file as the source. It may seem obvious in retrospect but that’s what happens when your codebase gradually evolves. When this happens, always try to find some time to stop, look back and ask yourself: “Is this code doing what it was originally made for?”.

Happy coding!


Read more from Untitled Kingdom:

0
0
Also tagged Digital Health

How much does it cost to develop a digital health app and why?

Klaudia Raczek

Klaudia Raczek

Marketer interested in startups. Building software for web & mobile apps for digital health, FemTech, IoT and digital transformation @ untitledkingdom.com

Klaudia Raczek

Klaudia Raczek

Marketer interested in startups. Building software for web & mobile apps for digital health, FemTech, IoT and digital transformation @ untitledkingdom.com

Feb 21, 2019 6 min read
Also tagged Business

What’s my purpose of running a business?

Also tagged Work Culture

Software company vs. digital product studio. What’s the difference? What questions should you ask?

Klaudia Raczek

Klaudia Raczek

Marketer interested in startups. Building software for web & mobile apps for digital health, FemTech, IoT and digital transformation @ untitledkingdom.com

Klaudia Raczek

Klaudia Raczek

Marketer interested in startups. Building software for web & mobile apps for digital health, FemTech, IoT and digital transformation @ untitledkingdom.com

Jan 07, 2019 9 min read